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ABSTRACT 


This  study  investigates  the  operational  implications 
of  the  differences  between  submarine  Target  Motion  Analysis 
(TMA)  experts.  A  submarine's  use  of  passive  sonar  provides 
uncertain  data  required  to  make  certain  decisions. 

This  experiment  presents  four  individual  scenarios  to 
three  submarine  TMA  experts:  Commanding/Executive  Officers 
(CO/XO) ,  Department  Heads  (DH) ,  and  Fire  Control 
Technicians  (FT)  .  Ten  individuals  from  each  expert  group 
volunteered  from  the  Groton,  CT,  and  Bangor,  WA,  submarine 
bases . 

A  between  subject  design  experiment  compared  the 
ranges,  range  envelopes,  time,  and  over  or  under 
estimations  of  range  to  the  contact  generated  by  each  group 
of  experts.  After  the  experiment  subjective  and  objective 
data  were  analyzed  in  order  to  determine  what,  if  any, 
differences  exist  between  the  three  different  experts. 

The  results  indicate  that  there  was  no  significant 
difference  between  experts.  Recommendations  address 
improvements  in  experiment  implementation  which  can  be 
integrated  into  future  studies  as  well  as  the  design  of 
improved  decision  aids. 
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EXECUTIVE  SUMMARY 


Submarines  cannot  see  underwater  so  the  crew  is  forced 
to  make  decisions  based  on  information  provided  by  the 
submarine's  sensors.  Passive  sensors  provide  information 
that  is  uncertain,  and  uncertainties  in  the  information 
displayed  on  a  submariner' s  computer  screens  ultimately 
affect  the  decisions  made  by  the  Commanding  Officer. 

Areas  of  uncertainty  (AOU)  refer  to  a  graphical 
display  tool  that  has  been  developed  in  order  to  place  more 
confidence  in  the  uncertain  data  obtained  by  passive 
sensors.  A  decision  aid  that  can  best  represent  AOUs  to  a 
variety  of  users  is  a  much  sought  after  tool.  A  more 
effective  decision  aid  can  be  developed  by  evaluating  the 
differences  between  different  submarine  subject  matter 
experts  (SMEs) . 

This  thesis  evaluates  the  differences  in  TMA 
performance  between  the  Commanding/Executive  Officers, 
Department  Heads,  and  Eire  Control  Technicians  with  regard 
to  how  they  construe  uncertain  information  when  displayed 
in  the  AOU  format.  No  significant  difference  was  noted 
between  the  three  expert  groups  regarding  range  accuracy. 
The  results  of  this  study  have  been  forwarded  to  the  Naval 
Undersea  Warfare  Center  (NUWC)  in  order  to  develop  AOU 
decision  aids  to  be  used  by  the  fleet. 
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1. 


INTRODUCTION 


Submarines  are  becoming  a  useful  tool  in  establishing 
safer  ocean  boarders  and  accomplishing  strategic 
objectives.  Many  aspects  of  the  submarine  service  have 
changed  over  the  past  107  years.  Advancements  in 

technology  have  altered  the  way  submariners  conduct  their 
business.  Many  nations  are  either  producing  or  purchasing 
more  submarines  in  order  to  reap  the  benefits  of  these 
"denizens  of  the  deep." 

A.  TARGET  MOTION  ANALYSIS  RESPONSIBILITIES 

Ultimately  the  responsibility  of  the  submarine  rests 
upon  the  shoulders  of  the  Commanding  Officer  (CO)  .  When  a 
submarine  is  engaged  with  a  threat,  either  surfaced  or 
submerged,  the  CO  will  typically  rely  upon  a  team  of 
individuals  comprised  of  the  Executive  Officer  (XO)  , 
Department  Heads  (DH) ,  and  Fire  Control  Technicians  (FT)  in 
order  to  maneuver  the  boat  without  incident.  While  a 
submarine  is  submerged  it  is  blind  with  the  exception  of 

information  received  by  the  Sound  Navigation  and  Ranging 
(sonar)  equipment.  As  technology  has  advanced  so  have  the 
means  by  which  we  can  interpret  sonar  data.  Even  highly 
sophisticated  equipment  provides  data  that  contains  some 
degree  of  uncertainty.  Data  uncertainty  is  generated  by 

many  sources  in  the  environment  and  the  submarine  itself. 
Making  decisions,  given  the  data  uncertainty,  is  a 
challenging  task  for  any  seasoned  submarine  crew. 

When  a  contact  of  interest  (COI)  is  initially 

detected,  a  section  tracking  party  will  be  stationed  in 

order  to  track  the  contact  in  a  more  detailed  manner  than 
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what  is  normally  assigned.  Submarine  crews  routinely 
conduct  just  enough  Target  Motion  Analysis  (TMA)  on  a 
contact  in  the  open  ocean  environment  to  safely  navigate 
the  ship  without  collision.  The  stakes  are  higher  when 
submerged  in  a  high  density  surface  environment  (lots  of 
contacts)  or  when  detecting  another  submerged  submarine; 
more  individuals  are  required  in  the  control  room  in  order 
to  safely  navigate  the  ship. 

B.  SOUND  NAVIGATION  AND  RANGING  CHARACTERISTICS 

Sonar  systems  listen  to  sounds  in  the  surrounding 
environment  and  report  any  noise  detections  to  an  operator. 
The  sonar  operator  will  then  interpret  the  reported  noise 
in  order  to  classify  the  source.  The  source  of  the  sound 
could  be  anything  from  biologies  (fish)  to  an  anchor  chain 
used  on  a  buoy  that  marks  a  traffic  lane  in  a  strait.  No 
matter  where  the  sound  originated  from  it  is  priority 
number  one  to  determine  if  the  sound  source  is  a  threat  to 
the  safety  of  the  submarine. 

A  submerged  submarine  presents  a  unique  environment 
for  sonar  operators  solving  range  estimates  to  other 
submerged  contacts.  When  provided  with  two  parameters 
(bearing  and  frequency) ,  sonar  operators  are  challenged  to 
generate  three  new  parameters  (course,  speed,  and  range) . 
These  parameters  are  used  to  classify  the  contact  and 
provide  solutions  to  targets. 

Data  provided  by  sonar  systems  are  distributed  to 

equipment  that  is  used  by  the  CO,  XO,  DHs,  and  enlisted 

specialists  such  as  FTs .  Each  of  these  individuals 

approach  a  submerged  contact  scenario  based  on  their  own 

intuitive  skills  that  they  have  been  taught  or  learned  by 
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experience  throughout  their  careers.  These  three  groups 
have  a  different  level  of  achieved  skill  and  when  working 
together  provide  an  excellent  combined  team  effort  in  order 
to  successfully  accomplish  the  many  difficult  tasks  of 
submarine  life. 

C.  AREA  OF  UNCERTAINTY 

Sound  travels  differently  in  water  than  it  does  in 
air.  Water  provides  a  difficult  challenge  for  sonar 
operators  when  trying  to  locate  the  source  of  sound. 
Background  noise  in  the  ocean  environment,  depth  of  the 
water,  and  even  the  composition  of  the  water  provide  an 
added  degree  of  uncertainty  with  regard  to  the  source 
location.  The  equipment  used  by  submarine  operators  and 
decision  makers  is  a  lot  better  than  that  used  in  the  past; 
however,  there  will  always  be  some  degree  of  uncertainty 
with  regard  to  the  information  that  is  provided  to  the 
operator . 

During  an  encounter  with  another  submerged  contact  a 
submarine  crew  must  adequately  deal  with  the  uncertainty  in 
the  data  in  order  to  safely  conduct  their  operations.  An 
Area  of  Uncertainty  (AOU)  is  generated  so  that  the 
submariners  can  have  an  idea  of  the  whereabouts  and 
direction  of  motion  of  the  submerged  contact.  AOUs  provide 
a  general  understanding  of  the  contact  while  the  crew  is 
working  to  determin  the  exact  location  of  the  source.  As 
more  time  elapses  and  a  better  understanding  of  the  contact 
develops,  the  AOU  will  shrink  in  size  and  provide  more 
benefit  to  those  trying  to  analyze  the  contact. 
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D.  ILLUSTRATIVE  EXAMPLE  OF  AN  AREA  OF  UNCERTAINTY 

Imagine  that  you  want  to  take  your  family  to  the  zoo 
in  order  to  see  the  new  lion  exhibit.  After  parking  the 
car  and  walking  to  the  zoo  entrance  you  hear  a  lion  roar. 
Without  using  any  maps  of  the  zoo  it  would  be  hard  to 
pinpoint  the  exact  direction  from  which  the  noise 
originated.  It  would  be  safe  to  assume  that  the  noise  came 
from  inside  the  zoo  so  the  AOU  would  be  the  area  of  the 
whole  zoo. 

After  entering  the  zoo  the  lion  roars  again.  This 
time  it  is  louder  and  more  directional.  If  you  were  to 
hear  the  noise  from  your  left  side  then  it  would  be  safe  to 
say  that  the  lion  is  not  on  your  right.  A  louder  roar  also 
indicates  that  the  exhibit  is  much  closer  than  it  was 
originally.  The  generated  AOU  for  the  lion  has  changed  in 
size  to  a  smaller  area  because  the  direction  to  the  noise 
is  more  discernable  than  it  was  in  the  parking  lot. 

After  walking  to  your  left  for  a  few  minutes,  and 
without  using  a  map  to  the  exhibit,  the  lion  lets  out 
another  roar.  This  time  the  noise  is  much  louder  and  even 
more  directional.  The  AOU  for  the  lion's  exhibit  becomes 
even  smaller  than  it  was  when  you  initially  entered  the  zoo 
and  we  can  pinpoint  almost  the  exact  location  from  which 
the  noise  came.  A  skilled  person  who  has  heard  many  roars 
from  lions  before  may  even  be  able  to  estimate  the  correct 
distance  to  the  exhibit. 

In  the  above  scenario,  some  sources  of  uncertainty  may 
make  it  more  difficult  to  determine  the  best  course  to  get 
to  the  lion  exhibit.  Factors  such  as  fog  in  the  parking 


lot,  construction  in  the  zoo,  or  even  human  limitations 
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like  hearing  loss  would  add  to  the  confusion  of  trying  to 
locate  the  exhibit.  The  fog  would  hide  the  size  of  the  zoo 
so  the  initial  AOU  would  have  been  much  larger.  Any 
construction  in  the  zoo  may  mask  the  lion' s  roar  and  make 
it  difficult  to  pinpoint  the  source.  Human  hearing  loss 
would  also  add  to  the  confusion  of  where  exactly  the  roar 
originated  and  all  of  this  uncertainty  would  have  produced 
larger  AOU  sizes  during  the  scenario  progression. 

E.  APPLICATION  OF  THE  AREA  OF  UNCERTAINTY 

During  a  section  tracking  party  evolution  there  are 
more  people  in  the  submarine  control  room  than  during 
normal  business  hours.  This  increase  of  submariners 

permits  a  more  effective  analysis  of  a  contact  by  allowing 
more  skilled  operators  to  analyze  the  sonar  data.  The 
initial  detection  of  a  contact  usually  is  the  cause  for 
stationing  the  section  tracking  party.  Detecting  a 

submerged  contact  from  longer  distances  allows  more  time  to 
station  the  section  tracking  party,  who  can  therefore  begin 
the  analysis  before  the  contact  becomes  too  close. 

Bearing  and  frequency  are  the  key  components  received 
by  the  sonar  equipment.  By  using  these  two  valuable  bits 
of  information,  contact  solutions  can  be  computed  to  gain  a 
better  understanding  of  how  the  contact  is  behaving.  As 
time  progresses  and  information  is  received  and  analyzed,  a 

better  understanding  of  what  the  contact  is  doing  will 

develop . 

Several  independent  stations  onboard  the  submarines 

work  to  analyze  contact  data.  A  geographical  plot  is  used 

to  generate  several  different  AOUs  during  the  encounter. 

Initially  the  AOU  will  be  large  because  of  the  increased 
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amount  of  uncertainty  with  regard  to  the  sonar  contact.  As 
the  scenario  progresses,  and  confidence  in  the  signal 
either  increases  or  decreases,  the  AOU  will  change 
inversely.  The  AOU  is  not  the  sole  source  of  a  contact's 
evaluation  but  it  does  provide  a  lot  of  insight  and  also 
provides  a  historical  view  of  what  the  contact  did  during 
the  encounter. 

As  several  AOUs  develop  over  a  period  of  time,  an 
understanding  of  the  contact's  motion  is  established.  For 
example,  if  you  stood  outside  the  lion  exhibit  and  closed 
your  eyes,  you  could  listen  to  the  roars  of  the  lion  very 
clearly.  If  initially  you  hear  the  lion  roar  to  your  left 
and  subsequently  hear  the  lion  roar  to  your  right,  you 
could  deduce  that  the  lion  is  traveling  from  left  to  right. 

A  contact  solution  is  generated  by  the  geographical 
plotter  based  on  observing  the  generated  AOU  and  is 
provided  to  the  XO.  Several  different  sources  of  contact, 
such  as  the  time  frequency  station  and  the  sonar 
supervisor,  evaluate  the  contact's  information  in  order  to 
generate  their  own  independent  solutions.  The  XO  is 
required  to  obtain  as  many  differently  sourced  contact 
solutions  as  possible  in  order  to  determine  the  best 
possible  solution  for  the  contact.  The  solution  provided 
by  the  geographical  plotter  based  on  AOU  will  be  assigned  a 
certain  level  of  confidence  by  the  plot  supervisor  and  the 
XO  will  take  this  confidence  into  account  while  generating 
his  own  solution.  The  combined  solution  generated  by  the 
XO  is  then  passed  to  the  CO  as  the  best  possible  solution. 
The  CO  will  add  another  level  of  scrutiny  and  will  base  his 
command  decision  on  this  refined  contact  solution.  The 
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whole  process  is  extremely  challenging  and  requires  skilled 
operators  in  order  to  effectively  conduct  target  motion 
analysis  (TMA)  operations. 

F.  PROBLEM  DEFINITIONS 

There  are  many  steps  involved  in  producing  a  solution 
that  has  a  high  confidence  level  to  the  CO.  The  AOU  is  a 
small  but  important  part  of  the  process.  Making  decisions 
given  uncertain  information  is  a  daily  event  onboard  a 
submarine.  More  accurate  range  estimates  can  be  determined 
if  an  AOU  is  utilized  by  submariners  during  the  TMA 
process.  During  a  submerged  contact  scenario,  observing 
the  AOU  changes  over  a  given  time  period  will  allow  the 
operator  to  accurately  estimate  the  contact's  range  and  to 
estimate  an  accurate  solution  to  the  COI . 

It  is  unknown  how  accurate  submarine  operators  are  at 
estimating  ranges  given  the  AOU  data.  There  are  generally 
three  different  recognized  expert  analysts  onboard  U.S. 
submarines  that  use  the  AOU.  To  make  an  assumption  that 
they  analyze  the  AOU  differently  is  logical.  It  is  also 
logical  to  assume  that  all  three  groups  (CO/XO,  DH,  and 
FTOW)  would  have  different  accuracies  once  they  generate  a 
range  estimate  based  on  the  AOU.  A  better  understanding  of 
the  differences  between  these  three  groups  of  individuals 
allows  for  further  research  into  how  to  possibly  display  an 
AOU  best  and  the  users  can  then  report  better  TMA  range 
estimates  when  they  are  presented  with  uncertain  data. 

G.  METHODOLOGY 

The  research  was  conducted  by  presenting  the  three 
different  groups  of  submarine  TMA  experts  with  several 
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scenarios  involving  AOUs.  During  each  computer  generated 
scenario  the  AOUs  shifted  around  the  monitor  to  simulate 
contact  motion.  Basing  their  contact  solution  on  the 
displayed  AOU,  each  participant's  solutions  were 
scrutinized . 

The  range  estimates  were  examined  through  extensive 
statistical  analysis  in  order  to  determine  if  there  was 
indeed  a  difference  in  accuracy  between  the  three  groups  of 
experts.  Any  significant  differences  in  accuracy  will 
provide  for  further  research  investigations.  The 
difference  between  the  participant's  solution  and  the 
scenario  truth  was  used  as  the  measure  of  effectiveness 
(MOE)  .  A  smaller  range  difference  indicates  that  the 
participant  was  more  accurate  than  if  they  had  generated  a 
large  range  difference. 
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II.  LITERATURE  REVIEW 


A.  TARGET  MOTION  ANALYSIS 

Target  motion  analysis  (TMA)  is  used  by  submarine 
operators  to  locate  and  track  contacts  in  submerged 
environments.  The  process  of  TMA  is  time  consuming; 
several  refined  solutions  for  a  given  contact  are  required 
before  a  section  tracking  party  obtains  an  accurate 
solution.  This  chapter  provides  background  information  on 
the  types  of  sonar  that  are  available,  the  TMA  process,  and 
the  different  causes  of  uncertainty  when  obtaining  data 
during  a  TMA  evolution. 

A  submerged  submarine  presents  a  unique  environment 
for  sonar  operators  solving  for  range  estimates  to  other 
submerged  contacts.  When  provided  with  two  parameters 
(bearing  and  frequency)  ,  sonar  operators  are  tasked  with 
the  difficult  job  of  accurately  producing  three  parameters 
(course,  speed,  and  range) .  Research  has  demonstrated  that 
sonar  operators  can  provide  more  accurate  range  estimates 
to  a  submerged  contact  if  they  are  given  a  graphical 
representation  of  the  uncertainty  surrounding  their  sonar 
solutions  (Kirschenbaum  and  Arruda,  1994;  Brolese,  2005; 
Brolese  and  Huf,  2006)  . 

1 .  Sound  Navigation  and  Ranging 

There  are  two  basic  types  of  sonar:  active  and  passive 

(L-3  Communications  SeaBeam  Instruments,  2000;  Liberty, 

2006)  .  Most  submarines  can  use  an  active  sonar  system  that 

transmits  a  pulse  of  sound  energy  into  the  ocean  and 

listens  for  a  returned  echo.  A  returned  echo  may  help  a 

submarine's  fire  control  tracking  team  locate  a  Contact  of 
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Interest  (COI) .  Active  sonar  typically  provides  a  more 
accurate  solution  for  the  contact;  the  drawback  to  using 
active  sonar  is  that  the  submarine  will  give  away  it's 
position  to  a  contact. 

More  often  than  not,  submarine  crews  will  track  their 
prey  using  passive  means  (Liberty,  2006;  Dry,  Lee,  Vickers, 
and  Huf,  2005)  .  This  is  done  by  using  different  sensors 
(or  arrays)  in  order  to  passively  obtain  contact 

information  while  emitting  a  minimal  amount  of  noise  into 
the  environment.  Using  passive  sonar  allows  submarines  to 
remain  quiet  and  still  formulate  a  solution  to  a  contact 
they  are  tracking.  The  submarine  tracking  party  can  only 
calculate  a  solution  based  on  the  information  that  is 
received  by  the  sensors.  While  tracking  a  contact  with 
passive  sonar  a  submarine  crew  is  limited  to  whatever  noise 
the  contact  emits.  Bearings  (By)  and  frequency  (f)  are  the 
two  main  parameters  detected  with  passive  sonar  and  are 
both  utilized  when  calculating  solutions  to  a  COI  (Coll, 

1994;  Bakos,  1995). 

2 .  Signal  Frequency 

Frequency  of  the  received  signal  can  be  used  to 
identify  the  source  of  the  signal,  estimate  a  range  to  the 
contact,  and  eventually  generate  an  independent  contact 

solution.  All  submarines  generate  a  unique  noise  signature 
that  is  susceptible  to  detection  by  a  submerged  sensor 

(Liberty,  2006) .  This  unique  signature  allows  the  crew  to 
classify  the  submarine.  Before  a  submarine  crew  deploys 
they  gather  intelligence  on  suspected  submerged  contacts 
they  may  detect  while  on  deployment.  Using  this 

intelligence,  an  estimate  of  the  COI's  maximum  speed  and 
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behavioral  patterns  can  be  postulated  by  the  watch  section. 
Speed  data  is  a  key  factor  in  determining  the  contact's 
solution  as  the  scenario  progresses. 

A  range  estimate  to  the  COI  can  be  calculated  by 
evaluating  several  parameters  regarding  the  local  marine 
environment,  then  using  these  parameters  to  predict  the 
distance  that  noise  could  travel  and  be  detected. 
Equipment  used  by  the  section  tracking  party  is  limited  by 
the  sensitivity  of  the  sensors  in  the  passive  sonar  arrays. 
By  understanding  the  limitation  of  the  passive  sonar 
sensors  a  prediction  can  be  made  regarding  the  effective 
range  at  which  any  contact  can  be  detected.  Frequency  data 
obtained  on  the  COI  may  also  be  used  to  determine  the 
maximum  range  of  a  contact.  High  frequencies  typically 
indicate  the  contact  is  closer.  Some  low  frequencies  can 
travel  extremely  long  distances  certain  water  temperatures. 
Higher  frequencies  suffer  too  much  attenuation,  over  long 
distances . 

3 .  Bearing 

Bearing  data  on  a  contact  is  obtained  from  the  ship' s 
sensors.  By  observing  contact  bearings,  for  a  given  time 
period,  a  bearing  rate  can  be  calculated.  For  example,  if 
an  initial  detection  on  a  submerged  contact  has  a  bearing 
of  090  and  ten  minutes  later  the  contact  has  a  bearing  of 
080,  then  the  calculated  bearing  rate  is:  left  one  degree 
per  minute.  Bearing  rates  are  continually  being  updated 
and  are  a  necessity  in  order  to  best  understand  what  the 
COI  is  doing.  A  high  bearing  rate  (five  degrees  per 
minute)  typically  indicates  the  contact  is  close;  a  lower 
bearing  rate  (one  degree  per  minute)  is  an  indication  that 
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the  contact  is  further  away.  However,  these  are  general 
rules  of  thumb  as  there  are  always  exceptions  regarding 
range  estimates  based  on  bearing  rates. 

Figure  1  shows  an  example  of  bearing  data  from  a 
submerged  contact.  When  a  submarine  crew  plots  bearing 
data  to  a  contact,  initially  there  is  a  large  number  of 
possible  solutions  for  the  COI .  Two  of  these  possible 
solutions  for  the  contact  are  depicted  by  SI  and  S2 
respectively.  Both  SI  and  S2  require  the  COI  to  travel  at 
different  speeds  and  ranges.  A  submarine  tracking  party 
will  use  a  variety  of  tools  and  equipment  to  analyze  the 
data  to  calculate  the  best  contact  solution.  The  amount  of 
data  obtained  on  a  given  contact  can  be  overwhelming. 

This  thesis  is  part  of  an  ongoing  experiment  by  the 
Naval  Undersea  Warfare  Center  (NUWC)  in  order  to  better 
understand  how  experts  view  AOUs.  The  rationale  for  this 
thesis  experiment  is  to  reduce  the  cognitive  load  on  the 
fire  control  tracking  team  by  examining  possible  displays 
and  identify  differences  in  performance  based  on 
experience.  Cunningham  and  Thomas  (2005)  provide  an 
excellent  source  for  more  in  depth  TMA  background 
information . 
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Figure  1.  Example  of  Bearing  Data. 

Figure  1  is  a  display  of  a  simplified  scenario  illustrating  the  nature 
of  TMA  bearings.  For  this  example,  ownship  is  taking  a  path  defined  by 
the  points  PI .  .  4  and  is  assumed  to  have  a  constant  velocity  and  have 
regular  bearings  sample  rate.  Each  point  is  equidistance  and 

represents  the  point  at  which  the  bearings  B1  .  .  4  were  taken.  These 
bearings  give  an  indication  of  the  direction  of  the  target  relative  to 
the  point  of  ownship  where  the  bearings  were  taken.  (From:  Cunningham 
and  Thomas,  p.  82,  2005). 


4 .  Uncertainty 

Unlike  the  surface  Navy  where  sailors  can  use  visual 
means  to  locate  contacts,  submariners  rely  upon  their 
equipment  to  detect  contacts.  This  reliance  upon  sonar 
imposes  restrictions  upon  the  degree  of  confidence  that  is 
placed  upon  the  received  data  (Dry,  et  al,  2005;  Schunn, 
Kirschenbaum,  and  Trafton,  in  press;  Brolese  and  Huf, 
2006)  .  The  ocean  environment,  submarine  equipment,  and  even 
the  submariners  themselves  generate  a  large  amount  of 
uncertainty  in  this  inherently  difficult  task.  Considering 
the  sources  of  uncertainty  permits  a  section  tracking  party 
to  generate  contact  solutions  in  which  the  assumptions  and 
potential  flaws  or  weaknesses  are  made  explicit. 
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Sound  travels  approximately  five  times  faster  in  water 
than  it  does  in  air  because  water  is  much  denser.  Sound  is 
also  more  susceptible  to  interference  in  water  because  the 
characteristics  of  the  fluid  can  change  rather  quickly  in 
an  ocean  environment.  The  density  of  the  water  is  affected 
by  pressure,  temperature,  and  salinity  (Liberty,  2006) . 
Signal  ambiguity  develops  because  sound  does  not  typically 
travel  in  straight  lines  (Kirschenbaum  and  Arruda,  1994)  . 

Temperature  and  salinity  can  change  rather  quickly 
because  of  ocean  currents  and  environmental  factors.  These 
factors  are  not  typically  predictable.  Pressure  changes 
are  the  most  predictable  because  it  changes  only  with 
depth.  The  varying  density  of  water  will  affect  the  speed 
and  path  in  which  sound  travels  through  water.  (Schmidt, 
Chayes,  Caress,  2006)  .  This  new,  unpredictable  path  the 
sound  traveled  through  the  water  increases  the  level  of 
uncertainty  with  regard  to  any  contact  solution  that  is 
generated . 

The  submarine  sensors  and  electronic  gear  used  in 
passive  sonar  systems  are  not  perfect.  After  a  signal  is 
detected  by  the  sonar  system  it  is  processed  by  electronic 
equipment  and  presented  to  an  operator.  A  small  degree  of 
uncertainty  is  generated  between  the  sensors  detecting  a 
signal  and  the  signal  being  presented  to  an  operator 
because  of  the  algorithms  used  by  the  computer 
(Kirschenbaum  and  Arruda,  1994) .  The  human  operator, 
similar  to  the  sensor,  has  inherent  flaws  in  judgment  and 
sensory  skills  required  for  analysis.  Even  when 
uncertainty  is  high  decisions  must  be  made.  Any  tool  that 
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could  assist  a  section  tracking  party  in  making  better 
decisions,  given  these  uncertain  conditions,  would  be  worth 
investigating . 

5.  Area  of  Uncertainty 

An  AOU  can  be  calculated  using  computer  algorithms 
based  on  environmental  parameters  and  contact  information. 
Figure  2  illustrates  an  example  of  a  possible  AOU  with  two 
different  colored  ellipses  generated  for  the  contact  in 
Figure  1.  The  inner  ellipse  indicates  a  probability  that 
is  less  than  the  probability  represented  by  the  inner  and 
outer  ellipses  combined.  For  example,  the  inner  ellipse 
could  represent  a  66%  probability  that  the  COI  is  located 
within  that  region  and  the  outer  ellipse  could  indicate  a 
95%  probability  the  COI  is  located  within  both  the  inner 
and  outer  ellipses  combined.  Research  is  ongoing  to 
determine  the  best  combination  of  ellipse  probabilities  in 
order  to  best  represent  uncertainty. 


Figure  2.  Possible  AOU  for  the  Bearing  Data  in  Figure  1. 
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During  each  ownship  maneuver  phase  of  the  contact 
evaluation,  the  AOU  will  change  based  on  subsequent  data 
received.  As  more  data  is  obtained  on  the  contact, 
unlikely  solutions  can  be  eliminated,  and  the  AOU  will 
shrink  in  size.  With  a  smaller  AOU,  the  process  of 
predicting  a  contact  solution  becomes  more  reliable.  If 
the  mission  is  to  launch  a  torpedo  or  to  go  to  periscope 
depth  the  AOU  can  provide  a  good  estimate  for  range  data  on 
the  contact  during  that  evolution. 

6 .  Area  of  Uncertainty  Research 

When  a  graphical  representation  of  uncertainty  is 
displayed  for  users,  the  degree  of  effectiveness  of  the 
displayed  uncertainty  varies  with  regard  to  the  difficulty 
level  (easy  or  hard)  of  the  scenario.  Large  differences  in 
the  performance  of  participants  between  easy  tasks  (low 
noise  scenarios)  and  harder  tasks  (high  noise  scenarios) 
have  been  observed  (Kirschenbaum  and  Arruda,  1994;  Brolese, 
2005)  .  When  comparing  the  use  of  uncertainty  ellipses  it 
was  demonstrated  that  the  aid  was  only  effective  during  the 
harder  scenarios. 

Kirschenbaum  and  Arruda  investigated  the  effects  of 
using  a  graphical  representation  of  uncertainty  to 
represent  the  probability  that  a  contact  is  contained 
within  the  ellipse  (Kirschenbaum  and  Arruda,  1994) . 
Brolese  extended  the  Kirschenbaum  and  Arruda  research  to 
investigate  what  method  of  uncertainty  representation  leads 
to  the  best  overall  performance  by  the  operator  (Brolese, 
2005)  . 

There  were  two  different  scenarios  presented  to  the 

participants  in  the  Brolese  study.  Five  different  ellipse 
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probabilities  were  used  in  both  scenarios.  Brolese 
measured  the  percentage  of  times  a  participant  was  able  to 
correctly  estimate  a  range  to  a  contact  for  all  ellipse 
percentages.  Specifically,  the  contact  had  to  be  contained 
within  the  range  limits  set  by  the  participant.  The  range 
estimates  reported  by  the  participants  using  the  95% 
ellipse  contained  the  COI  approximately  60%  of  the  time. 
The  research  demonstrated  that  the  95%  ellipse  was  more 
effective  in  supporting  participants  when  estimating 
overall  range  data  to  contacts. 

Figure  3  supports  that  in  difficult  scenarios,  unlike 
easier  scenarios,  there  was  indeed  a  significant  difference 
in  the  accuracy  of  range  estimates  provided  by  the 
participants.  In  the  more  difficult  scenarios  both  the  50% 
and  95%  AOU  produced  approximately  similar  results  with 
regard  to  minimizing  absolute  range  errors.  The  95% 
ellipses  had  the  most  pronounced  effect  on  aiding  the 
participants'  performance  during  both  the  difficult  tasks 
and  easier  tasks  in  that  it  produced  the  lowest  absolute 
range  errors  and  more  correct  range  estimates  on  average. 
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Figure  3.  Ellipse  Condition  versus  Absolute  Error. 

(From:  Brolese,  2005) . 

7 .  This  Study 

The  primary  aim  of  the  present  study  was  to  continue 
to  explore  the  findings  of  Kirschenbaum  and  Arruda  (1994), 
and  Brolese  and  Huf  (2006)  ,  by  examining  the  effect  of 
expertise  of  users  regarding  displayed  uncertainty. 
Kirschenbaum  and  Arruda  recommend  further  research  in  the 
perceptual  and  cognitive  elements  used  by  submarine  experts 
and  their  use  of  uncertainty  ellipses  to  better  understand 
the  role  of  expertise  in  range  estimations  (Kirschenbaum 
and  Arruda,  1994)  . 

The  AOU  provided  to  participants  in  the  present  study 
was  similar  to  the  95%  ellipses  shown  in  previous  studies. 
During  the  course  of  the  present  study,  Brolese  and  Huf 
reported  additional  research  that  casts  doubt  upon  which 
ellipse  AOU  (50%  or  95%)  provides  the  most  benefit  to  the 
user  (Brolese  and  Huf,  2006)  .  These  findings  may  be  of 


50%  75%  95%  99%  Inf  Abs 
Ellipse  Condition 


military  significance  to  the  Australian  Navy  because  their 
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submarine  operators  are  only  provided  a  95%  ellipse. 
Brolese  and  Huf  demonstrated  that  "50%  ellipses 
consistently  lead  to  more  correct  and  accurate  proximity 
decisions  than  the  currently  utilized  95%  ellipse"  (Brolese 
and  Huf,  2006,  p.  5)  . 

Given  the  conflicting  findings  between  the  two  studies 
it  was  decided  that  the  use  of  an  ellipse  to  depict  a  95% 
AOU  would  not  be  inappropriate  for  the  present  study.  All 
participants  were  subject  to  the  same  conditions  during  the 
experiment  and  the  purpose  of  the  study  was  to  determine 
any  significant  differences  in  performance  between  the 
different  groups  of  subjects. 

By  administering  scenarios  to  operators  with  three 
different  levels  of  expertise  within  the  submarine 
community,  the  mission  is  to  determine  if  there  are  any 
significant  differences  in  range  estimates  between 
subjects.  The  different  levels  of  expertise  are: 
commanding/  executive  officers,  department  heads,  and  fire 
control  technicians.  All  three  groups  of  experts  responded 
to  the  same  scenarios  and  their  contact  solutions  were 
analyzed  to  determine  if  one  particular  expert  group 
provided  more  accurate  ranges. 

B.  DECISION  AIDS 

The  ability  of  submariners  to  interpret  and  reason 
with  decision  aids  is  increasingly  important  as  technology 
advances.  Some  of  the  cognitive  workload  may  be  mitigated 
by  the  use  of  well-designed  decision  aids.  An  aid  that 
reduces  mental  workload  and  allows  a  user  to  obtain  needed 


information 

"at 

a  glance" 

is 

a  much 

sought 

after  tool 

(Hutchins , 

19  9  6; 

Larkin 

and 

19 

Simon, 

1987)  . 

Cognitive 

scientists  and  cognitive  systems  engineers  are  among  those 
who  are  skilled  at  understanding  the  way  in  which  humans 
interact  with  their  environment. 

Diagrammatic  representation  and  reasoning  is  an  area 
of  research  that  is  focused  on  how  humans  or  machines  can 
represent  information  using  diagrams  and  solve  a  problem  or 
answer  a  question  while  using  those  diagrams  (Narayanan, 
1997).  Cognitive  scientists  apply  diagrammatic  reasoning 
to  understand  the  interaction  of  three  complex  factors: 
cognitive  and  perceptual  skills  of  the  user,  graphical 
properties  of  the  external  representation,  and  the 
requirements  of  the  specific  task  being  undertaken  (Peebles 
and  Cheng,  2003)  . 

Regarding  depicting  AOUs,  a  number  of  investigations 
have  reported  how  cognitive  and  perceptual  skills  of  the 
user,  graphical  properties  of  the  external  representation, 
and  the  requirements  of  the  specific  task  being  undertaken 
have  affected  the  user  (Kirschenbaum  and  Arruda  1994; 
Brolese  2005;  Brolese  and  Huf  2006)  .  The  focus  of  these 
investigations  was  to  develop  a  tool  specifically  designed 
to  represent  uncertainty  with  the  goal  of  supporting 
submariners  in  their  decision  making  process. 

Submariners  operate  in  an  extremely  complex 
environment.  The  size  of  the  crew  is  limited  so  there  are 
typically  several  tasks  assigned  to  each  member  of  a  watch 
section.  Multitasking  is  an  advantage  only  when  the  user 
can  efficiently  perform  each  task.  As  the  number  of  tasks 
increase  so  does  the  amount  of  information  that  needs  to  be 
assessed.  Designers  of  graphical  interfaces  seek  to 
consider  how  different  quantities  are  encoded  and  how  to 
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balance  the  cost  of  familiar  representations  with 
computationally  advanced  representations  in  order  to  enable 
a  user  to  efficiently  analyze  the  data.  Guidelines  for 
further  development  of  visual  displays  and  graphical 
interfaces  have  been  developed  by  research  (Peebles  and 
Cheng,  2003)  . 

1 .  Principles  of  Decision  Aiding 

Decision  theorists  argue  that  in  order  for  optimal 
decision  making  to  take  place  a  thorough  analysis  of  all 
data  and  hypotheses  must  occur;  these  evaluations  typically 
involve  complex  equations  and  extensive  calculations  in 
order  to  arrive  at  an  optimal  decision.  Analysis 

strategies  employed  by  decision  makers  generally  involve 
sophisticated  mental  computations  and  a  lot  of  data 
(Hutchins  1996,  Dry,  Lee,  Vickers,  and  Huf,  2005) . 

Graphics,  by  virtue  of  the  presentation  of  an 
integrated,  synthesized  representation  of  the  critical 
data,  allow  users  to  omit  time  consuming  steps,  such  as 
calculations  and  estimations,  in  their  evaluation  of  data. 
For  example,  during  a  contact  evaluation  in  a  submarine 
scenario,  several  range  estimates  to  a  COI  are  calculated. 
These  calculations  are  typically  cognitively  challenging, 
as  well  as  time  consuming.  When  a  graphical  depiction  of 
range  is  available  this  eliminates  the  need  to  calculate 
range  and  thus  frees  up  an  operator' s  mental  capacity  for 
other  tasks  (Hutchins  1996;  Dry,  et  al,  2005). 

Beyond  range  calculations,  a  large  amount  of  data 

(speed,  course,  bearing,  range,  and  bearing  ambiguity  when 

using  a  towed  array)  need  to  be  analyzed  by  a  submariner 

during  the  TMA  process.  In  addition  to  the  processing  of 
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data,  working  memory  is  required  for  other  tasks.  An 
Officer  of  the  Deck  (OOD)  uses  working  memory  to  maintain  a 
list  of  actions  taken  by  the  section  tracking  party  during 
an  evolution  and  for  predicting  possible  scenario  outcomes. 

Predicting  is  a  difficult  cognitive  task  that  depends 
heavily  on  working  memory  (Wickens,  Lee,  and  Liu,  2004) .  In 
order  to  predict  accurately  one  must  consider  current 
conditions,  future  conditions,  and  then  mentally  simulate, 
using  the  decision  maker' s  mental  model  to  determine  the 
expected  outcome.  When  our  mental  resources  are  tasked  to 
the  limit  (maximum  capacity)  or  overrun  with  several 
concurrent  tasks  our  ability  to  predict  rapidly 
deteriorates.  A  graphical  display  that  includes  a  summary 
of  critical  data  would  decrease  the  cognitive  workload 
required  to  synthesize  the  data  and  free  up  the  mental 
capacity  of  the  OOD  for  other  tasks. 

In  the  process  of  reducing  the  mental  workload 
required  of  a  submariner,  well  designed  displays  may  also 
produce  the  additional  benefit  of  minimizing  the  amount  of 
human  error  in  decision  making.  The  operation  of  these 
systems  in  the  fleet  is  the  true  test  of  how  well  the 
principles  of  decision  aiding  have  been  observed  during  the 
research  and  development  phase  of  production. 

Newly  developed  systems  will  be  judged  on  their 
ability  to  minimize  the  amount  of  incompatibility  placed 
between  the  system  and  the  user  (Hutchins,  1996) .  Research 
on  the  way  humans  perceive  displayed  information  and  make 
decisions  on  that  information  is  important  in  developing 
decision  aids  that  minimize  mental  workload,  provide  needed 
information  "at  a  glance",  and  minimize  human  error. 
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2. 


Data  Fusion 


The  quality  of  an  aid  and  the  way  the  human  utilizes 
the  information  provided  by  the  aid  are  two  factors  that 
ultimately  determine  the  user's  performance  during  a  task. 
Specifically  how  much  trust  the  user  places  in  the  aid,  the 
way  in  which  information  is  presented  to  the  user,  time 
pressure,  and  the  level  of  risk  involved  affect  the  overall 
utilization  of  an  aid.  The  process  of  data  fusion  involves 
input  from  several  sources,  each  of  which  provides 
information  at  a  different  level  of  abstraction  (Bisantz, 
Finger,  Seong,  and  Llinas,  1999). 

Information  provided  from  a  data  fused  plot  is 
presented  to  the  user  through  the  computer  interface. 
These  interfaces  require  training  the  user  before  the  user 
can  fully  understand  what  is  presented.  An  interface  with 
multiple  levels  of  information  presented  in  a  fashion  that 
is  easily  understood  by  the  user  can  save  time  and  effort 
when  formulating  solutions  in  a  complex,  uncertain 
environment.  Specifically,  ease  of  recognition  is  directly 
related  to  how  explicit  and  implicit  information  is 
represented  (Larkin  and  Simon,  1987)  . 

3.  Factors  Regarding  an  Aid  (Trust  and  Presentation) 
a.  Trust 

Trust  in  an  advanced  automated  system  does  not 
come  easily.  The  amount  of  trust  placed  in  an  aid  will 
affect  a  user's  reliance  on  the  system  (Sheridan,  1988)  . 
Empirical  results  suggest  that  the  strategy  employed  by 
users  may  be  affected  by  the  trust  in  that  system.  For 
example,  when  participants  required  financial  advice  from 
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an  automated  system,  confidence  in  the  system  declined  when 
poor  advice  was  given  (Lerch  and  Prietula,  1989)  .  A  new 
strategy  for  most  users  was  to  not  use  the  system  at  all 
because  of  the  lost  trust. 

b.  Presentation 

The  utility  of  an  aid  is  determined  by  how  the 
information  is  presented.  Past  research  has  focused  on  how 
to  display  uncertainty  so  that  the  true  probabilities  of 
the  data  can  be  presented  to  a  user  (Kirschenbaum  and 
Arruda,  1994;  Andre  and  Cutler,  1998;  Brolese  and  Huf, 
2006;  Brolese,  2005)  .  The  research  by  Andre  and  Cutler 
(1998)  and  by  Kirschenbaum  and  Arruda  (1994)  demonstrate 
that  different  visual  representations  of  uncertainty 
resulted  in  a  measurable  difference  in  user  accuracy. 
Brolese  and  Huf  (2006)  and  Brolese  (2005)  built  on  the 
Kirschenbaum  and  Arruda  (1994)  research  by  determining  how 
to  specifically  represent  uncertainty  visually  in  order  for 
the  user  to  achieve  the  most  accurate  result. 

In  the  research  by  Andre  and  Cutler,  when  no 
visual  representation  was  provided  to  the  user,  the 
participants  appeared  to  forget  that  there  was  any  degree 
of  uncertainty  involved  in  the  experiment.  Kirschenbaum 
and  Arruda  (1994)  demonstrated  that  when  verbal 
representations  of  uncertainty  were  presented  to  their 
participant  performance  was  less  accurate  than  when  an 
elliptical  ring  representing  uncertainty  was  displayed. 
These  findings  suggest  that  it  is  worthwhile  to  continue 
investigating  how  best  to  visually  represent  uncertain 
information . 


A  well-designed  decision  aid  will  support  the 
user  by  reducing  the  demand  on  the  limited  cognitive 
processing  capability  of  human  beings  (Dry,  et  al .  ,  2005)  . 

Because  of  our  limited  processing  abilities,  a  decision 
maker  may  fail  to  remember  critical  data,  overlook 
important  information,  make  decisions  too  hastily,  and 
possibly  make  decisions  that  are  not  correct  (Hutchins, 
1996)  . 


4 .  Graphical  Displays 

Graphical  displays  visually  present  measured 
quantities  of  data  by  providing  combined  points,  lines,  a 
coordinate  system,  numbers,  symbols,  words,  shading,  and 
color  to  a  user.  Graphics  are  instruments  that  support 
complex  reasoning  about  the  quantitative  information 
provided.  Well-designed  data  graphics  are  typically  the 
simplest,  and  at  the  same  time,  the  most  powerful  form  of 
information  presentation  available  (Tufte,  2001). 

Graphical  displays  should  show  data  in  a  format  that 
best  supports  the  user's  needs,  induce  the  viewer  to  think 
about  the  substance  of  what  they  are  viewing  rather  than 
the  means  of  getting  it,  avoid  distorting  what  the  data 
represent,  encourage  the  eye  to  compare  different  pieces  of 
data,  and  be  closely  integrated  with  the  statistical 
descriptions  of  a  data  set  (Tufte,  2001)  .  The  more  closely 
the  presented  data  represent  the  needed  information  to  the 
user,  the  better  the  user  can  effectively  use  the  data. 
Larkin  and  Simon  (1987)  believe  the  advantage  of  a  good 
diagram  is  computational;  that  is,  the  representation  of 
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information  is  not  necessarily  based  on  the  amount  but  on 
the  ability  of  the  user  to  successfully  compute  accurate 
results . 

There  is  limited  research  on  the  impact  of  displayed 
uncertainty  on  the  performance  of  decision  makers  (Brolese 
and  Huf,  2006;  Kirschenbaum  and  Arruda,  1994) .  Research  on 
how  graphical  displays  can  represent  data  the  most 
effectively  is  imperative  for  a  technologically  advanced 
society . 

5.  Advantages  of  Graphic  Displays 

Recent  technological  advancements  have  led  to  dramatic 
increases  in  computational  speed  and  sophistication  of  the 
graphical  display  capabilities  of  computers.  New 
technological  formats  will  interact  with  task  structure  and 
improve  situational  awareness  and  task  performance  of 
submariners  (Kirschenbaum  and  Arruda,  1994).  Research  in 
required  to  determine  how  these  new  improvements  will 
affect  a  user's  ability  to  analyze  data  and  make  decisions. 

While  the  processing  capabilities  of  computers  have 
increased  the  processing  capabilities  of  the  user  have  not. 
If  display  designers  do  not  take  into  account  the 
limitations  of  the  user,  then  it  is  unlikely  that  the  true 
potential  of  the  new  technology  will  be  realized  (Dry,  et 
al .  ,  2005).  Technological  improvements  with  respect  to 
automation  do  not  necessarily  translate  into  benefits  for 
the  user  (Roth,  Patterson,  and  Mumaw,  2002) . 

Representation  aiding  is  where  direct  manipulations 
and  graphical  techniques  make  abstract  information  more 
concrete  in  complex  and  dynamic  domains  (Bennett,  Toms,  and 
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Woods,  1993)  .  Analog  to  digital  conversions  are  required 
in  order  for  data  to  be  presented  in  these  new  formats. 
Configurable  properties  of  the  data  displayed  can  be  either 
structural  or  dynamic.  The  structural  and  dynamic 
properties  of  a  display  describe  the  behavior  or  movement 
of  graphical  elements  with  respect  to  other  elements 
(Bennett,  Toms,  and  Woods,  1993)  . 

6.  Cognitive  Engineering 

"Cognitive  engineering  is  an  interdisciplinary 
approach  to  the  development  of  principles,  methods,  tools, 
and  techniques  to  guide  the  design  of  computerized  systems 
intended  to  support  human  performance"  (Roth,  Patterson, 
and  Mumaw,  2002,  p.  164;  Roth  and  Woods,  1988)  .  The  main 
cognitive  areas  of  concern  are  problem  solving,  judgment, 
decision  making,  attention,  perception,  and  memory.  The 
goal  of  cognitive  engineering  is  to  develop  interactive 
systems  that  are  easy  to  learn,  and  easy  to  use,  with  the 
end  result  being  improved  user  performance  (Roth, 
Patterson,  and  Mumaw,  2002)  . 

Cognitive  engineering  advocates  that  users  and  the 
tasks  they  perform  should  be  the  central  focus  for  system 
design  specification.  The  design  of  systems  is  viewed  as  a 
means  to  create  a  tool  that  assists  users  in  their  decision 
making.  This  approach  benefits  the  user  because  the 
computer-based  tools  and  aids  will  be  designed  to  solve  the 
appropriate  problem.  Cognitive  engineering  provides 
opportunities  for  computational  technology  to  increase  the 
potential  to  facilitate  and  augment  human  cognitive 
activities,  e.g.,  advanced  data  visualization  techniques 
used  in  decision  making  (Roth,  Patterson,  and  Mumaw,  2002)  . 
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Considerable  time  and  money  have  been  spent  developing 
cognitive  tools  to  improve  human  interfaces  with 
technologically  advanced  software.  A  specific  tool  was 
designed  to  support  cognitive  engineers  in  developing 
software  which  models  and  documents  the  Cognitive  Task 
Analysis  (CTA) .  The  Computer-Aided  Cognitive  Systems 

Engineering  (CACSE)  tool's  benefit  is  that  the  design  of  a 
Decision  Support  System  (DSS)  will  be  a  direct  result  of 
the  cognitive  issues  and  relationships  determined  during 
the  CTA.  "System  developers  using  the  CACSE-supported 
process  would  deliver  fully  functional  DSSs  that  embody 
solutions  to  the  cognitive  demands  of  the  domain  and 
provide  dramatically  improved  joint  (human  +  DSS)  decision¬ 
making  effectives  -  a  critical  component  of  information 
dominance"  (Potter,  Elm,  Roth,  and  Woods,  2001,  p.  2)  . 

7 .  Psychological  Implications  for  Submarine  Display 

Design 

a.  Perceptual  Factors  Influencing  Display 
Design 

Perceptual  factors  must  be  considered  by 
engineers  in  the  designing  of  displays.  Several  factors 
may  make  it  difficult  for  a  user  to  effectively  use  a 
display  to  make  decisions.  When  using  grey  scales  for 
displayed  information,  the  background  may  determine  the 
brightness  of  the  different  shades  used.  These  varying 
levels  of  contrast  may  be  distracting  to  the  user  and  may 
also  cause  additional  mental  fatigue. 

When  using  color  in  display  design,  there  are 
implications  as  to  which  colors  are  located  adjacent  to 
each  other  (Dry,  et  al,  2005)  .  For  example,  red  and  blue 
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should  not  be  used  in  close  proximity  because  of  a 
phenomenon  called  chromatic  aberration.  Chromatic 

aberration  is  the  difference  in  wavelengths  between  two 
colors  in  close  proximity  where  the  user  cannot  focus  on 
the  two  colors  at  the  same  time  (Bradley,  Zhang,  and 
Thibos,  1989)  . 

b.  Cognitive  Factors  Influencing  Display  Design 

Designing  displays  requires  that  human  cognition 
be  evaluated  in  order  to  increase  the  usefulness  of  the 
display  (Wickens,  Lee,  and  Liu,  2004)  .  Several  factors 
such  as  attention  and  short-term  memory  limitations  require 
consideration  when  designing  displays.  Humans  can  only  pay 
attention  to  limited  amounts  of  information  for  limited 
amounts  of  time  (Wickens,  Lee,  and  Liu,  2004;  Dry,  et  al, 
2005)  . 

The  short-term  memory  limitations  of  the 
display's  user  must  also  be  considered.  Too  much 

information  on  a  display  will  limit  the  users  effectiveness 
with  regard  to  the  decisions  that  are  made  based  on  the 
information  presented.  The  amount  of  information  that  can 
be  stored  in  short-term  memory  is  believed  to  be  five  to 
nine  items,  varying  according  to  complexity,  presentation 
sequence,  and  length  of  time  the  information  must  be 
remembered  (Dry,  et  al,  2005) . 

8 .  Quantity  of  Information  Presented  in  an  Aid 

Research  provides  an  understanding  of  the  quantity  of 
information  required  by  experts  while  making  decisions.  It 
has  been  suggested  that  it  is  not  necessarily  the  quantity 
of  information  used  by  experts,  but  the  ability  of  the 
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expert  to  focus  on  what  information  is  relevant  (Ettenson, 
Shanteau,  and  Krogsad,  1987;  Shanteau,  1992b).  Researchers 
and  decision  aid  developers  must  take  into  account  the 

cognitive  factors  of  the  human  when  designing  their  work. 
Quality,  not  the  quantity,  of  information  should  be 
emphasized  by  designers  who  are  building  aids  for  decision 
makers . 

C.  EXPERTISE  AND  DECISION  MAKING 

Judgment  and  decision-making  (J/DM)  is  an  important 
topic  of  study  for  human  factors  researchers.  Two  specific 
fields  to  study  within  the  topic  of  J/DM  are  experts  and 
expert  systems  (Shanteau  and  Stewart,  1992) .  In  the  fields 
of  decision  making,  planning,  and  problem  solving, 

expertise  has  been  one  of  the  most  difficult  concepts  to 
understand,  capture,  and  quantify  (Serfaty,  MacMillan, 
Entin,  and  Entin,  1997)  .  Even  though  experts  are  perceived 
by  non-experts  as  "those  people  with  all  the  answers"  and 
are  the  ones  we  trust  in  their  specific  field  they  are 
certainty  not  perfect.  In  fact,  experts  have  cognitive 

limits,  individual  characteristics,  and  preferred 
strategies  that  are  common  amongst  their  everyday  practices 
(Shanteau,  1988;  Wickens,  Lee,  Liu,  and  Becker,  2004) . 
Experts  certainly  are  not  infallible  and  their  complex 
behaviors  are  just  one  reason  they  are  so  difficult  to 
study  (Edwards,  1992). 

Many  experts  employ  decision  aids.  It  would  be  a 

mistake  to  study  only  the  expert  and  not  the  aids  they  use 
and  the  amount  of  information  they  require.  By 
understanding  the  way  experts  make  decisions,  and  the 
information  required  for  them  to  make  these  decisions, 

30 


researchers  hope  to  develop  expert  systems  that  "are  based 
on  codification  of  the  knowledge  and  decision  rules  of 
experts"  (Shanteau  and  Stewart,  1992,  p.  101). 

The  domain  of  military  command  and  control  often 
involves  complex  decision  making  situations,  uncertain 
information,  and  high-tempo  tactical  and  operational 
environments.  In  these  situations  there  is  typically  more 
than  one  correct  answer  to  a  problem.  One  characteristic 
that  differentiates  the  novice  from  the  expert  is  the 
timeliness  of  decisions  made.  Experts  appear  to  be  better 
at  making  decisions  with  information  that  is  limited  and 
ambiguous,  entails  a  high  degree  of  uncertainty,  and  is 
within  a  multidimensional  battlespace.  The  ability  to  make 
these  decisions  has  been  effectively  labeled  as  battle 
command  decision-making  expertise  (Serfaty,  et  al . ,  1997). 

1 .  Why  Study  Expert  Decision  Making 

There  are  many  different  definitions  for  "experts." 
An  expert  is  "one  with  the  special  skill  or  knowledge 
representing  mastery  of  a  particular  subject."  (Webster's, 
1993)  This  definition  provides  the  rationale  for  trying  to 
determine  if  there  is  indeed  a  difference  between 
recognized  experts  in  the  field  of  submarine  TMA.  Shanteau 
and  Stewart  (1992)  state  that  there  are  at  least  three  good 
reasons  why  research  on  expert  decision  makers  is 
important:  generalizing  of  differences  between  naive 
subjects  and  experts;  advancement  of  expert  systems  through 
examination  of  those  running  the  system;  and,  experts  are 
interesting  in  their  own  right. 
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a. 


Generality 


It  is  not  enough  to  assume  there  is  a  difference 
between  recognized  experts  in  a  field  of  study,  but  to 
discover  what  exactly  the  differences  are.  During  the 
course  of  a  study  if  similar  behaviors  are  observed  between 
participants  than  an  argument  for  the  universality  of  the 
observed  behavior  can  be  made.  The  same  goes  for  like 
behavior  between  participants  in  similar  studies.  For 
example,  if  the  majority  of  experts  and  novices  make 
decisions  in  a  similar  way  then  we  can  establish  guidelines 
that  represent  behavior  for  all  participants. 

Analyses  of  experts  provide  an  important  means  of 
establishing  generality  (Shanteau  and  Stewart,  1992).  If 
there  is  a  difference  between  the  novice  and  expert 
participants  then  new  guidelines  need  to  be  created.  These 
guidelines  will  be  broader  in  scope  and  will  account  for 
all  observed  participants  in  both  studies  and  in  the 
working  environment. 

b.  Expert  Systems 

"Expert  systems  are  based  on  the  codification  of 
the  knowledge  and  decision  rules  of  experts"  (Shanteau  and 
Stewart,  1992,  p.  101) .  Studying  experts  is  to  capture  the 
cognitive  strategies  they  employ  and  use  this  understanding 
to  develop  tools  to  assist  other  decision  makers  (Shanteau 
and  Stewart,  1992) .  Systems  that  fall  under  this  category 
of  "tools"  are  designed  to  assist  experts  in  their  decision 
making  or  to  make  decisions  that  are  comparable  to  those 
made  by  their  human  counterparts. 
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The  design  of  an  expert  system  involves  an  in- 
depth  study  of  how  experts  make  their  decisions.  The 
analysis  of  experts  will  give  us  a  greater  understanding  of 
when  and  where  these  systems  can  be  used  (Shanteau,  1992a) . 
Expert  systems  should  have  most  of  the  compiled  knowledge 
that  experts  have.  The  knowledge  and  strategies  employed 
by  decision  makers  can  be  utilized  to  better  represent  data 
in  expert  systems.  Experts  and  novices  use  data 

differently  so  the  way  in  which  data  is  presented  is  a 
challenge.  These  systems  should  be  designed  in  a  way  that 
allows  for  flexibility  in  the  processes  employed  to  make 
decisions.  Human  experts  are  flexible;  a  rigid  system 
would  best  be  used  to  simulate  novices  (Shanteau,  1989) . 

c.  Intrinsic  Interest 

Many  researchers  are  intrigued  by  domain  experts 
and  want  to  know  more  about  how  they  make  decisions 
(Shanteau  and  Stewart,  1992,  p.  102)  .  If  it  were  indeed 
true  that  the  best  decisions  are  made  only  by  experts  then 
it  may  be  assumed  that  the  novices  would  want  to  emulate 
experts.  As  a  junior  officer  (novice  submarine  ship 
driver)  this  author  always  wanted  to  be  like  the  senior 
officers  (expert  ship  drivers) .  I  was  always  interested  in 
their  thought  processes  during  exercises  and  how  they 
produce  accurate  solutions  for  COI  through  the  TMA  process. 

The  modern  military  relies  on  the  judgments  of 
expert  decision  makers  to  maintain  a  free  state.  Surface 
and  sub-surface  ship  drivers,  war  planners,  physicians,  and 
even  individual  members  of  a  platoon  are  expected  to  make 
expert  decisions.  These  demands  lead  researchers  to  many 
unanswered  questions  such  as:  How  good  are  these  expert's 
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decisions?  Can  their  decision  making  skills  be  improved? 
What  aids  can  be  utilized  to  assist  their  decision  making 
process  ? 

2 .  Challenges  in  Examining  Expert  Performance 

Identifying  experts  is  the  first  challenge  in 
conducting  an  experiment  in  this  field  of  study  (Edwards, 
1992)  .  Several  approaches  can  be  used  to  determine  who 
qualifies  as  an  expert  (e.g.,  credentials) .  It  may  be  that 
those  in  a  particular  domain  should  define  the  experts 
(Shanteau,  1992a),  or  that  it  takes  one  to  know  one 
(Edwards,  1992) . 

Once  you  have  determined  who  the  experts  are,  the 
second  challenge  is  to  gain  access  to  them  in  order  to 
study  their  decision  making  skills  (Edwards,  1992)  .  In 
order  for  experts  to  remain  experts  they  must  continually 
be  involved  in  the  field  in  which  they  work.  This  amount 
of  upkeep  may  change  drastically  between  different  fields 
of  study  but,  "If  you  don't  use  it  you  lose  it." 

The  next,  and  last,  challenge  is  to  specify  the 
measure  of  performance  used  to  analyze  the  differences 
between  participants  (Edwards,  1992) .  All  of  the 

challenges  discussed  above  may  be  overcome  with  some  well 
considered  planning  prior  to  a  given  study. 

3.  Self-Efficacy 

Self-efficacy  is  defined  as  the  confidence  level  in  an 
individual's  ability  to  execute  courses  of  action  or  to 
achieve  a  desired  outcome  in  the  face  of  adversity 
(Bandura,  1997.)  A  low  level  of  confidence  will  adversely 

affect  the  expected  performance  of  an  individual  during 
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competition  (Woodman  and  Hardy,  2003;  Parfitt  and  Pates, 
1999) .  There  are  several  theories  that  explain  this 
behavior.  Bandura  suggests  that  those  individuals  with 
high  self-efficacy  -  the  belief  that  one  can  achieve  what 
one  sets  out  to  do  -  are  confident,  effective,  and 
generally  more  successful  than  those  individuals  with  low 
self-efficacy  (Bandura,  1997). 

Bandura  suggests  that  self-efficacy  judgments  are 
derived  from  four  sources:  performance  accomplishments; 
vicarious  experiences;  verbal  persuasion;  and  emotional 
arousal  (Bandura,  1997) .  It  is  the  cognitive  appraisal  of 
the  four  sources  of  information  that  will  either  increase 
or  decrease  an  individual's  level  of  self-efficacy  and 
subsequent  confidence. 

4 .  Cognitive  Limitations  of  Experts 

Intelligent  people  have  great  difficulty  judging 
probabilities,  making  predictions,  and  otherwise, 
attempting  to  cope  with  uncertainty  (Kahneman,  Slovic,  and 
Tversky,  1982).  These  challenging  aspects  of  decision 
making  problems  are  a  key  explanation  for  why  humans  rely 
on  judgmental  heuristics  (Slovic,  Fischhoff,  and 
Lichtenstein,  1985;  Kahneman,  et  al .  ,  1982;  Wickens,  et 

al .  ,  2004;  Dry,  et  al . ,  2005).  Although  these  heuristics 

are  a  functional  tool  used  by  the  human  brain  to  compensate 
for  limited  cognitive  processing  ability  (Hutchins,  1996), 
they  can  lead  to  systematic  biases  and  errors  in  judgment 
(Kahneman,  et  al .  ,  1982)  .  These  heuristics  (or  mental 

rules  of  thumb)  may  explain  both  the  high  and  low  levels  of 
performance  observed  when  experts  make  decisions  (Shanteau, 
1988;  Shanteau  and  Stewart,  1992). 
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Conducting  TMA  on  individual  contacts  is  a  demanding 
task.  In  the  submarine  community  there  are  several  courses 
on  what  is  called  "mental  gym."  This  headache  inducing 
"gymnasium"  involves  the  application  of  recognized  formulas 
to  a  specified  scenario  in  a  limited  amount  of  time  without 
using  a  calculator.  This  work-out  is  the  backbone  of  why 
submariners  conduct  ship  operations  safely. 

5.  Characteristics  of  Experts 

Experts  within  similar  domains  may  display  similar 
psychological  characteristics  (Shanteau,  1988).  Experts 
within  a  given  domain  may  experience  similar  schools  or 
training.  All  submarine  officers  attend  similar  schools 
during  their  careers.  The  thought  process  and  analysis 
patterns  between  experts  are  similar;  however,  individual 
experiences  sharpen  the  skill  differently.  The  submariner 
TMA  characteristics  between  experts  are  domain  specific  and 
will  be  similar  because  of  the  training  requirements  in 
their  specified  field.  This  section  contains  a  brief 
description  of  some  psychological  traits  relevant  to  the 
experts  in  this  study.  A  more  complete  list  can  be 
reviewed  in  Shanteau  (1987)  . 

Experts  generally  have  highly  developed  perceptual/ 
attention  abilities  (Shanteau,  1988).  Eew  studies  have 
shown  how  experts  perceive  uncertain  data  and  how  they 
produce  effective  decisions  from  those  data.  It  takes  time 
to  perform  both  the  studying  and  practicing  portion  to  a 
particular  trade.  Any  individual  who  has  been  practicing  a 
trade  for  an  extended  period  of  time  may  qualify  as  an 
expert  in  that  field.  With  practice  experts  become  able  to 
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extract  information  that  the  non-experts  just  do  not  notice 
to  be  relevant  (Shanteau,  1988). 

Experts  have  a  sense  of  what  is  relevant  and 
irrelevant  (Shanteau,  1988)  .  When  experts  are  given  too 
much  information  they  should  be  able  to  sift  through  the 
data  and  rely  on  the  relevant  information  only.  Experts 
can  determine  relevant  information  because  of  the  insight 
they  have  gained  through  experience  in  their  fields 
(Shanteau,  1992b). 

Experts  have  an  ability  to  simplify  complex  problems 
(Shanteau,  1988) .  Breaking  down  the  fundamentals  of  a  game 
and  making  complex  problems  more  simple,  experts  can  then 
make  better  informed  decisions.  In  the  game  of  chess, 
recognized  experts,  have  superior  pattern  recognition 
capabilities  that  allow  them  to  perform  better  (De  Groot, 
1965)  . 

Experts  are  able  to  handle  adversity  better  than  non¬ 
experts  (Shanteau,  1988)  .  Experts  work  better  under  stress 
and  make  better  decisions.  Given  a  scenario  where  there  is 
uncertainty,  the  experts  must  determine  relevant 
information  (Shanteau,  1992b)  to  minimize  mistakes  in 
judgment . 

6.  Strategies  of  Decision  Making  Experts 

While  studying  domain  experts,  many  different 
strategies  have  been  observed  that  lead  to  better 
decisions.  Experts  are  able  to  make  adjustments  during  the 
decision  making  process  and  generally  avoid  large  mistakes 
(Shanteau,  1988) .  Domain  experts  use  their  experience  to 
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continuously  make  decisions  in  an  evolving  situation 
several  times  over  the  course  of  a  problem. 

For  example,  as  soon  as  expert  Texas  Hold-Em  poker 
players  realize  that  their  hand  is  no  longer  the  best  one, 
they  will  fold  it  right  away  to  save  themselves  money.  The 
poker  hand  these  "Pros"  initially  started  with  may  have 
been  the  best  one  at  the  start  of  the  round  but  by  the  time 
all  the  cards  are  showing  on  the  table,  their  hands  may  be 
the  worst.  In  this  example,  they  must  develop  new  decisions 
every  time  another  card  is  exposed.  They  will  produce 
solutions  that  lead  to  the  least  error  or  will  save/make 
them  the  most  money.  "When  you  feel  that  you  are  beaten,  it 
is  time  to  toss  your  pretty  cards  into  the  muck.  It  may 
seem  silly  to  think  that  folding  may  be  a  key  in  your  poker 
progression,  but  saving  chips  that  should've  been  lost  is 
the  exact  same  as  winning  chips"  (Carlisle,  2006) . 

7.  Is  More  Information  Better  for  Expert  Decision 
Makers? 

A  study  regarding  the  amount  of  information  required 
by  experts  while  making  decisions,  conducted  by  Ettenson, 
Shanteau,  and  Krogstad  (1987),  provides  an  interesting 
approach  to  quantifying  the  amount  of  information  used. 
Their  study  suggested  that  it  was  not  necessarily  the 
quantity  of  information  used  by  experts,  but  the  ability  of 
the  expert  to  focus  on  what  information  was  relevant. 
Three  groups  of  auditors  were  required  to  made  judgments  on 
thirty-two  individual  cases  based  on  eight  dimensions  of 
accounting  information.  Results  indicated  that  all  three 
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groups  relied  upon  three  factors  out  of  the  eight  in  order 
to  make  their  judgments,  less  than  half  of  the  factors  that 
were  available  to  each  group. 

The  expert  groups  relied  most  heavily  upon  one  single 
factor  while  other  cues  had  secondary  impacts.  There  was 
no  single  cue  that  was  dominant  for  the  novice 
participants.  In  conclusion,  this  study  demonstrated  that 
choosing  to  use  less  information  was  not  a  cognitive 
limitation  but  an  indicator  that  the  experts  were  able  to 
focus  on  more  relevant  information.  Novices  appeared  to 
require  more  information  (Ettenson,  et  al .  ,  1987;  Shanteau, 
1991) .  Although  each  participant  was  given  a  wealth  of 
information,  much  of  it  was  not  required  and  did  not  add  to 
the  performance  of  the  three  groups . 

In  a  later  study,  Shanteau  states  that  "the  amount  of 
information  used  does  not  reflect  degree  of  expertise; 
however,  the  type  of  information  used  does"  (Shanteau, 
1992b,  p.  2)  .  Shanteau  also  states  that  there  is  indeed  a 
difference  between  mid-level  and  advanced  experts  because 
the  advanced  experts  are  better  at  evaluating  which 
information  is  the  most  relevant  (Shanteau,  1992b) .  A 
question  for  further  research  is  to  evaluate  the 
differences  in  decisions  made  between  different  levels  of 
expertise . 

In  support  of  Shanteau' s  work,  Omodei,  McLennan, 
Elliott,  Wearing,  and  Clancy  (2005) ,  state  that  more 
information  is  not  necessarily  better,  especially  given  an 
adverse  environment  with  uncertain  conditions.  The 
consensus  is  still  the  same  when  expert  decision  makers  are 
confronted  with  an  environment  that  is  "time-pressured. 
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distributed,  dynamic  command  environment  characterized  by 


uncertainty"  (Omodei,  et  al . ,  2005,  p.  39). 


8 .  Battle  Command  Decision-Making 

It  is  important  to  understand  how  an  expert  approaches 
a  decision-making  problem  to  better  understand  what  we 
observe  in  the  laboratory.  The  concept  of  "mental  models" 
(Serfaty,  et  al .  ,  1997),  is  used  to  further  explain  how 
experts  step  through  a  decision  making  experience. 


In  Figure  4,  a  five-step  process  is  depicted  in 
detail.  Mental  models  are  an  internal  interpretation  of 
what  we  experience  in  the  real  world.  As  decision  makers 
gain  experience  they  ought  to  proceed  through  each  step 
with  a  higher  degree  of  certainty.  Initially  we  would 
expect  the  expert's  pattern-indexed  memory  to  construct  a 
better  mental  model  of  the  situation. 
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Figure  4.  Five  Step  Process  for  "Mental  Model"  Analysis 

(From:  Serfaty,  et  al . ,  1997.) 


In  step  1,  the  expert  may  exhibit  characteristics  of  a 
recognition-primed  decision  by  generating  an  initial  model 
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of  what  they  have  observing  by  recalling  past  experiences. 
In  the  next  step,  the  expert  formulates  the  right  questions 
to  ask  in  order  to  reduce  uncertainty  in  his/her  initial 
assumptions.  This  may  be  done  in  a  time-constrained 
environment . 

In  step  3,  the  expert  will  now  produce  a  model  that  is 
described  as  "rich"  and  "dynamic  in  space  and  time."  This 
is  the  mental  model  that  the  expert  will  use  to  produce 

several  different  outcomes  in  step  4  and  develop  a  plan  of 
action  in  step  5.  The  number  of  different  outcome  models 
is  dependent  on  the  experience  level  of  the  expert.  As 

experience  is  gained  in  real  world  situations  experts  can 

produce  many  more  possible  outcomes  and  have  more  possible 
plans  from  which  to  choose. 

Experts  keep  an  extensive  store  of  specific 

experiences  in  memory,  supported  by  high-level  principles 
taught  in  a  classroom  or  on  the  job,  that  allow  them  to 
quickly  develop  a  rough,  still  incomplete  mental  model  for 
a  new  situation  (Serfaty,  et  al . ,  1997)  .  These  models  are 

typically  built  on  sound  principles  learned  by  the  expert 
over  time.  On  any  platform  in  the  service,  those 

individuals  making  decisions  would  expect  some  positive  or 
negative  feedback  from  individuals  supporting  them.  If  an 
experts  mental  model  is  wrong  than  he/she  may  require  some 
feedback  in  order  to  transform  their  model  into  a  more 
accurate  one . 
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Ill .  METHODOLOGY 


The  purpose  of  this  research  was  to  determine  if  there 
is  a  difference  in  decision  making  expertise  between  three 
different  groups  of  submarine  TMA  subject  matter  experts 
(SMEs)  when  using  a  graphical  display  that  depicts  the  AOU 
for  the  scenario.  The  Java  applicated,  AOU  study  was  used 
to  simulate  a  submerged  submarine  scenario  that  involved 
several  contacts  of  interest  (COI) .  Each  participant  was 
required  to  formulate  the  best  possible  solution, 
specifically  range,  for  one  selected  contact  using  the 
displayed  AOU.  The  end  result  was  to  be  either  a 
satisfactory  firing  solution  or  coming  to  periscope  depth. 

The  experiment  included  two  scenarios,  (i.e., 
targeting  and  navigation) ,  and  two  color  schemes  for  each 
scenario.  One  scenario  had  a  bright  center  and  dark  edge, 
the  other  had  a  dark  center  and  a  bright  edge.  All 
scenarios  had  multiple  contacts  with  at  least  one 
overlapping  AOU. 

Administering  the  four  individual  scenarios  to  each  of 
the  thirty  participants  (ten  from  each  SME  group)  ,  the 
hypothesis  that  there  is  a  difference  between  the  three 
SMEs  can  be  tested.  This  chapter  describes  the  process 
used  to  simulate  the  four  scenarios,  experimental  design, 
participants,  apparatus,  procedure,  and  data  collection. 

A.  EXPERIMENTAL  DESIGN 

The  experimental  task  was  to  evaluate  participants  who 
observe  a  simulation  of  a  submerged  submarine  which 
involved  AOUs  for  several  contacts.  Each  participant  was 
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to  formulate  the  best  possible  solution  for  a  specific  COI 
using  the  designated  AOU.  If  there  is  a  difference  between 
experts  it  would  be  because  of  any  confounding  variables. 

Two  different  scenario  backgrounds  were  presented  to 
each  participant.  In  the  first  scenario  each  participant 
was  to  use  the  information  presented  by  the  AOU  to  evaluate 
the  COI  and  report  when  they  felt  their  solution  was 
accurate  enough  to  man  battle  stations  and  attack.  In  the 
second  scenario  each  participant  was  required  to  use  the 
information  provided  by  the  AOU  to  evaluate  the  COI  and 
determine  when  it  was  safe  for  the  submarine  to  proceed  to 
periscope  depth  (PD)  . 

Each  of  the  scenarios  required  extensive  calculations 
and  application  of  the  decision  making  process  employed  to 
conduct  TMA  by  the  participant.  The  only  data  provided 
were  the  uncertainty  ellipses  displayed  in  the  specified 
AOU.  Each  scenario  (attack  and  PD)  had  two  different 
simulations  available  for  a  total  of  four  different 
scenarios.  Each  participant  engaged  in  all  four  scenarios 
as  well  as  a  practice  scenario. 

B.  PARTICIPANTS 

A  total  of  30  participants  volunteered  for  the 

experiment.  All  were  either  U.S.  Naval  officers  or 

enlisted  submarine  members.  Two  sites  were  visited  in 

order  to  obtain  enough  individuals  for  each  group  of  SMEs. 

Three  CO/XO,  eight  DH,  and  ten  ETs  were  participants  in 

Groton,  CT  while  all  other  participants  were  observed  in 

Bangor,  WA.  Groton  is  known  as  the  "submarine  capital  of 

the  world"  and  is  the  home  for  many  of  the  schools  relating 

to  the  submarine  service.  The  administering  officer  for 
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this  study  coordinated  times  for  conducting  the  experiment 
that  coincided  with  the  participants  schedules. 

The  CO/XO  participants,  on  average,  were  ten  years 
older  than  the  FT  participants  and  five  years  older  than 
the  DH  participants.  The  CO/XO  participants  were  also  the 
most  educated  of  all  participants  with  80%  of  them  having 
received  a  Master's  Degree. 

1 .  FT  Participants 

FT  participants  were  selected  using  two  criteria. 
First,  each  participant  had  to  be  a  second  class  petty 
officer  or  above.  Second,  each  participant  had  to  have 
been  an  instructor  at  a  FT  school.  The  reason  for  these 
criteria  was  to  ensure  that  the  participants  from  this 
group  had  enough  experience  in  the  field  of  TMA  to  be 
considered  an  Expert.  FT  participant  ages  ranged  from  26 
to  37  years,  with  the  average  age  being  33.7  years. 

2 .  DH  Participants 

The  Submarine  Officer  Advanced  Course  (SOAC)  was  the 
main  source  for  the  DH  participants.  Upon  graduation  from 
SOAC  these  Lieutenants  proceed  to  the  fleet  to  become 
Navigator,  Weapons,  or  Engineering  officers  onboard 
submarine  platforms.  The  DH  participants  from  SOAC  had  yet 
to  serve  as  an  actual  DH  on  a  submarine.  Several  DH 
participants  had  already  served  as  a  DH  on  a  submarine.  DH 
participant  ages  ranged  from  26  to  37  years,  with  the 
average  age  being  29.3  years. 


45 


3.  CO/XO  Participants 

The  CO/XO  participants  were  selected  using  one 
criterion.  The  CO/XO  participants  had  to  have  graduated 
from  the  Pre-Executive  Officer  School  (PXO)  .  Two  of  the 
CO/XO  participants  recently  graduated  from  the  PXO  course 
while  the  rest  of  the  CO/XO  participants  were  stationed  on 
submarines  or  had  completed  their  submarine  tour  as  a  CO  or 
XO.  CO/XO  participant  ages  ranged  from  34  to  44  years, 
with  an  average  age  of  39  years. 

C .  APPARATUS 

1 .  Laptop  Computer 

The  test  apparatus  consisted  of  a  Pentium  II  2GHz 
laptop  computer.  The  laptop  computer  was  placed  on  a 
standard  sized  desk  with  normal  lighting  conditions. 
Figure  5  depicts  a  demonstration  of  how  a  participant  may 
have  conducted  the  experiment.  The  exact  location  for  each 
participant  varied.  Participants  observed  the  scenarios 
from  their  work  offices,  hotel  rooms,  or  in  one  case,  their 
own  home . 
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Figure  5.  Example  of  the  Experimental  Setup. 

2 .  Surveys 

Four  different  surveys  were  completed  by  each 
participant.  All  of  the  surveys  were  designed  by  this 
author  and  his  advisor  with  the  exception  of  the  NASA  Task 
Load  Index  (TLX)  survey. 

a.  Biographical  Data  Survey  (Appendix  A) 

The  goal  of  this  survey  was  to  collect  basic 
information  such  as  rank,  rate,  age,  etc. 

b.  Scenario  Survey  (Appendix  B) 

The  goal  of  this  survey  was  to  collect 
participant  solutions  for  the  scenarios  and  to  determine 
what  characteristics  influenced  his  problem  solving. 
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c.  NASA  Task  Load  Index  (Appendix  C) 

The  TLX  is  a  subjective  workload  assessment  tool. 
It  is  compromised  of  six  subscales  that  measure  Mental 
Demand,  Physical  Demand,  Temporal  Demand,  Effort, 
Performance,  and  Frustration. 

d.  Post-Scenario  Questionnaire  (Appendix  D) 

The  goal  of  this  questionnaire  was  to  gather 
information  that  may  have  affected  the  decision  making  of 
the  participants  during  the  experiment.  The  information 
obtained  from  these  questionnaires  will  be  used  when 
designing  future  submarine  displays. 

D .  PROCEDURE 

1 .  Surveys 

Upon  arrival,  participants  were  required  to  read  and 
sign  the  consent  form  in  APPENDIX  F.  The  protocol  used  to 
perform  the  testing  is  in  APPENDIX  E.  In  order  to  ensure 
each  participant  experienced  the  same  situation  the  written 
protocol  form  in  APPENDIX  E  was  used  by  the  administering 
officer.  Participants  were  provided  an  overview  of  the 
study  as  well  as  being  informed  that  their  data  would  be 
compared  with  two  other  groups  of  participants  representing 
two  other  groups  of  experts.  The  researcher  explained  that 
a  95%  ellipse  for  a  given  contact  meant  that  the  contact 
had  a  probability  that  it  was  located  within  that 
particular  ellipse  95%  of  the  time  and  outside  the  ellipse 
5%  of  the  time. 

Participants  then  observed  the  practice  scenario  to 
familiarize  themselves  with  the  format  and  to  provide  them 
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with  examples  of  ellipses.  The  administering  officer  used 
Appendix  E  in  order  to  fully  explain  the  format  of  the  GUI 
display  and  controls.  Participants  were  given  as  much  time 
as  necessary  to  ask  questions  and  ensure  they  were 
comfortable  with  the  display  format. 

Once  a  scenario  began  it  could  only  be  stopped  when 
the  participant  conducted  the  TMA  and  produced  a  solution 
for  the  COI .  If  the  participants  did  not  manually  stop  the 
scenario  by  selecting  the  stop  button  (see  Figure  10 
ellipse  C)  then  the  screen  would  go  blank  and  he  would  be 
required  to  report  his  solution  (this  happened  six  times) . 

Appendix  B  contains  the  four  scenario  examples.  The 
scenarios  were  presented  in  a  partially  randomized  manner. 
The  order  was  limited  to  a  total  of  eight  different 
sequences.  Scenarios  One  and  Three  were  similar  with  the 
exception  that  Three  was  skewed  228  degrees.  Scenarios  Two 
and  Four  were  similar  with  the  exception  that  Four  was  were 
skewed  62  degrees.  To  prevent  the  participant  from 
noticing  a  similarity  between  any  skewed  scenario  pair,  no 
paired  scenarios  were  presented  back-to-back. 

Upon  completion  of  each  scenario,  participants  filled 
out  Appendices  B  and  C.  Each  Scenario  Survey  requested  a 
Range,  Range  +/-,  Bearing,  Bearing  +/-,  Course,  and  Course 
+/-.  Several  questions  pertained  to  the  weight  individuals 
placed  upon  certain  factors  during  their  decision  making. 
For  example,  on  a  scale  from  one  to  ten  (ten  being  the  most 
important)  how  much  did  the  safety  of  the  ship  weigh  in 
your  decision? 

As  stated  above,  the  NASA  Task  Load  Index  (Appendix  B) 
was  completed  after  each  scenario.  The  same  sheet  of  paper 
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was  used  for  all  four  scenarios.  When  a  participant  placed 
a  mark  on  the  scale  he  would  use  the  scenario  number  as  a 
mark.  For  example,  if  the  participant  felt  a  high  level  of 
Mental  Demand  for  scenario  Three  then  he  would  mark  a  three 
on  the  scale  close  to  the  high  end  of  the  scale.  On  a 
subsequent  scenario,  if  the  participant  felt  the  same  level 
of  mental  demand  then  he  would  place  that  scenario  number 
at  the  same  place  on  the  scale. 

Upon  completing  all  four  scenarios  and  all  of  the 
surveys  attached  to  those  scenarios,  the  participants 
completed  the  Post  Scenario  Survey  (Appendix  D)  .  The  Post 
Scenario  Survey  assessed  the  overall  experience  from  each 
participant  for  all  four  scenarios.  These  questions 
focused  on  the  confidence  level  of  each  participant  in 
using  an  AOU  to  assess  range  estimates  and  the 
participant's  experience  with  regard  to  TMA  decision 
making . 

2 .  Scenarios 

The  process  of  generating  scenarios  was  extensive. 
The  same  format  used  by  Kirschenbaum  and  Arruda  (1994)  was 
used  in  the  current  research.  A  meeting  between  the 
author,  Susan  Kirschenbaum,  Wendy  Berube,  and  Sandie  Grage 
was  conducted  at  NUWC  several  months  before  the  experiment 
was  conducted.  The  JAVA  coded  simulation  was  provided  by 
Ms.  Berube  and  the  displays  were  produced  by  Ms.  Sandie 
Grage.  Susan  Kirschenbaum  provided  the  background 
information  displayed  within  the  scenarios  (AOU  color  and 
information  format) . 
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a. 


JAVA  Simulation  Code 


The  Java  simulation  code  consists  of  three  files: 
Main. java,  Aou JFrame . j ava,  and  ImagesFrame . j ava .  Main. java 
is  the  class  that  is  called  first  and  instantiates  the 
AouJFrame  class.  Once  the  AouJFrame  is  instantiated,  all 
the  components  (panels,  text,  labels,  buttons,  etc)  for  the 
graphical  user  interface  (GUI)  are  displayed  and 
initialized.  The  AouJFrame  file  also  includes  the  methods 
for  opening  the  output  file  once  the  enter  button  is 
selected  and  writing  output  data  to  it  when  the  stop  button 
is  selected.  Lastly,  this  file  contains  methods  to  write 
the  ownship  and  target  position  data  which  corresponds  to 
the  displays  which  depict  the  AOUs  for  the  targets.  The 
ImagesFrame . j ava  file  first  creates  a  background  thread  to 
improve  efficiency  and  then  displays  the  AOU  displays 
sequentially  every  10  seconds.  The  Java  class  MediaTracker 
includes  built-in  functions  which  handle  displaying  images. 

b.  Displays 

The  four  scenarios  used  during  the  research  were 
generated  from  two  initial  scenarios.  Each  initial 
scenario  was  skewed  either  228  or  62  degrees  for  the  odd 
and  even  scenarios  respectively  to  produce  a  total  of  four 
scenarios.  Ms.  Sandie  Grage  and  some  military  advisors 
generated  the  geometries  for  the  two  scenarios.  These 
scenarios  were  created  by  a  hypothetical  periscope  and 
targeting  scenario  whiteboard  discussion.  Once  the 
scenarios  were  generated,  the  information  (such  as  target 
initial  bearing,  range,  course,  speed,  and  ownship 
maneuvers)  was  given  to  a  technician  who  coded  the 
scenarios  in  a  TI-04  simulator. 
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The  scenarios  were  run  through  the  TI-04  system 
and  the  outputted  data  were  collected.  Data  extracted  from 
the  collected  output  were  placed  into  a  functional  database 
and  then  a  Matlab  script  was  created  to  extract  information 
for  ownship,  target,  and  AOU  data.  A  state  estimation 
algorithm.  Nodestar,  was  used  to  calculate  contact 
solutions  for  the  scenarios.  A  Matlab  script  was  used  to 
convert  the  extracted  information  and  draw  the  display 
figures.  The  displays  were  drawn  and  depicted  ownship 
position,  each  target's  position,  each  target's  AOU,  and 
range  rings . 


c.  Scenario  Generation 


To  run  the  Java  application,  a  Java 
Development  Kit  (SDK)  was  installed  on  a  laptop 
All  files  for  the  program  were  stored  in  a  folder 
access  before  and  after  the  scenarios  were  run. 
each  scenario  was  run,  the  participant  number 
scenario  was  entered  into  the  file  name  section 
6.  For  example,  "001TarYelNov25DH"  would  be  the 
for  the  first  participant,  targeting  scenario 
yellow  centered  AOU,  administered  on  the  25^*^  of 
to  a  department  head  SME . 


Software 
computer . 
for  easy 
Before 
and  the 
of  Figure 
file  name 
with  the 
November, 
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Figure  6.  Mission  Selector  for  the  AOU  Study. 

To  begin  each  scenario,  after  entering  the  file 
name  and  selecting  a  mission,  the  administering  officer 
would  select  the  enter  button  in  Figure  6.  The  scenario 
would  begin  and  display  a  new  display  every  ten  seconds. 
When  a  satisfactory  solution  for  the  COI  was  determined  by 
the  participant,  he  stops  the  scenario  by  selecting  the 
stop  button  in  Figure  10,  ellipse  C.  When  the  stop  button 
was  selected  the  following  information  was  saved  in  a 
specified  file  location:  scenario,  subject  number,  start 
time,  and  stop  time. 

d.  Scenario  Description 

A  total  of  four  scenarios  were  generated  for  the 
present  research.  Appendix  B  contains  the  four  handouts 
given  to  the  participants  before  each  of  the  four 
scenarios.  The  Yellow  Centered  Targeting  Mission  was 
scenario  One.  The  Yellow  Centered  Navigation  Mission  was 
scenario  Two.  The  Red  Centered  Targeting  Mission  was 
scenario  Three.  The  Red  Centered  Navigation  Mission  was 
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scenario  Four.  Although  there  were  multiple  AOUs  in  each 
scenario,  each  participant  was  instructed  to  analyze  only 
the  AOU  assigned  to  the  COI . 

Each  scenario  included  a  total  of  ninety 
individual  displays.  The  GUI  formatted,  JAVA  coded  program 
presented  each  display  to  the  participant  in  a  specified 
order  that  simulated  the  motion  of  all  contact's  AOUs.  All 
contact  AOUs  originated  in  the  center  of  the  screen  and 
within  the  first  minute  of  each  scenario  the  AOUs  would 
steadily  migrate  out  to  a  realistic  location.  Each 
participant  was  told  that  they  must  wait  for  the  first  two 
minutes  of  each  scenario  (which  included  12  slides)  before 
they  could  begin  trusting  the  AOU  data  locations. 

Eigure  7  is  an  example  of  a  yellow  centered  AOU 
display.  Figure  8  is  a  red  centered  AOU  display  that  is 
similar  to  the  display  in  Figure  7  with  the  exception  that 
it  is  skewed  sixty-two  degrees  to  produce  a  different  even 
scenario.  In  Figure  7  there  are  five  individual  AOUs 
present.  Each  AOU  is  labeled  SI  through  S5  respectively 
and  each  AOU  has  several  different  probability  ellipses 
available  for  evaluation.  Different  colors  are  used  to 
indicate  probabilities  that  the  contact  is  located  inside 
that  AOU.  The  red  ellipse  indicates  there  is  a  95% 
probability  that  S4  is  located  within  the  red  and  yellow 
rings  combined.  The  yellow  ellipse  represents  a  66% 
probability  that  S4  is  located  in  the  yellow  ring  only. 
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Figure  7.  Yellow  Centered  AOU  Scenario  Display. 


Figure  8.  Red  Centered  AOU  Scenario  Display. 

Relative  motion  was  simulated  by  the  Java  coded 
program  displaying  subsequent  displays  during  a  scenario. 
Figure  7  and  Figure  9  depict  examples  of  subsequent 
displays.  Figure  7  would  be  displayed  first  and  ten 
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seconds  later  Figure  9  would  be  displayed.  When  reviewing 
the  two  displays  notice  the  difference  in  S4's  location. 
In  Figure  7,  the  right  edge  of  S4's  AOU  is  positioned  at 
the  ten  thousand  yard  range  ring.  In  Figure  9  the  right 
edge  of  S4's  AOU  is  now  positioned  half  way  between  the  ten 
thousand  and  five  thousand  yard  range  rings.  This 

difference  between  Figure  7  and  Figure  9  would  simulate  S4 
moving  toward  the  center  of  the  screen. 


Figure  9.  Yellow  Centered  AOU  Scenario  Display  Subsequent  to 

the  Jpeg  in  Figure  7 . 

e.  Practice  Scenario 

A  practice  scenario  allowed  participants  to 
familiarize  themselves  with  the  controls  of  the  program  and 
the  simulation  format.  The  practice  scenario  included  an 
in-depth  explanation  of  the  controls  and  setup  of  the 
program.  This  practice  scenario  was  similar  to  that  used 
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by  Kirschenbaum  and  Arruda  (1994)  and  was  essentially  a 
walk  through  by  the  administering  officer. 


Figure  10  is  a  screenshot  of  the  practice 
scenario  that  was  viewed  by  each  participant.  The 
simulated  ownship  marker  during  all  scenarios  was  at  the 
center  of  the  screen.  Four  range  rings,  valued  at  five 
thousand  yards  apiece,  for  a  total  of  twenty  thousand  yards 
surrounded  the  ownship  marker.  Ellipse  A,  in  Figure  10,  is 
an  enlarged  ownship  marker  at  the  center  of  the  screen. 

Ellipse  B  in  Eigure  10  is  an  enlarged  display  of 
ownship  speed  and  course,  as  well  as  a  COI  solution 
generated  by  a  fire  control  computer  is  shown.  The 
generated  fire  control  solution  was  explained  to  each 
participant  as  a  solution  that  would  be  generated  by  a  fire 
control  technician  onboard  a  submarine  while  underway. 
This  generated  solution  was  not  to  be  considered  as  truth 
but  more  of  an  assistant  to  the  participant. 

Ellipse  C  in  Eigure  10  is  an  enlarged  stop 
button.  When  the  participant  was  comfortable  with  his 
solution  for  the  COI  he  was  instructed  to  click  on  the  stop 
button  using  the  mouse  pointer.  Each  participant  was 
toldto  remember  his  COI  solution  before  selecting  stop 
because  when  the  stop  button  was  selected  the  screen  would 
disappear . 
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Figure 


10.  Example  of  the  Practice  Scenario  Screenshot  with 
Individual  Sections  Outlined  and  Magnified. 


E.  DATA  ENTRY  AND  FORMATTING 

The  data  (Appendix  B)  needed  to  be  transformed  prior 
to  analysis.  Data  formatting  for  expertise,  time,  and 
range  are  described  in  the  following  sub-sections.  The 
corresponding  data  are  provided  in  Appendices  M  through  P. 

1 .  Expertise 

SMEs  were  grouped  based  on  their  level  of  expertise. 
The  SMEs  were  FTs  (Group  One)  ,  DHs  (Group  Two)  ,  and  COs/XOs 
(Group  Three) .  With  ten  participants  in  each  group, 

therewere  30  data  points  for  each  scenario.  The  three 
different  groups  of  expertise  were  recorded  in  the  expert 
column  of  Appendices  M  through  P. 
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2. 


Time 


The  length  of  time  each  participant  took  to  complete  a 
scenario  required  two  conversions  in  order  to  analyze  the 
data  correctly.  The  first  conversion  required  the  output 
of  the  Java  program  to  be  rounded  down  to  the  nearest  ten 
seconds.  This  rounding  was  done  because  each  new  display 
was  presented  in  ten  seconds  increments  during  the 
scenario.  If  a  participant  selected  the  stop  button  at 

00:10:45  (ten  minutes  and  forty  five  seconds)  then  he  based 
his  solution  on  the  display  at  00:10:40  (ten  minutes  and 

forty  seconds) .  The  second  conversion  was  required  to 
convert  a  00:00:00  formatted  time  into  a  number  with  three 
decimal  places.  For  example,  00:07:40  would  convert  to 
7.667  minutes.  This  converted  time  is  recorded  in  the  time 
column  of  Appendices  M  through  P. 

3 .  Range 

The  range  data  lists  what  each  participant  believed 
the  correct  range  to  the  COI  was  at  the  time  he  stopped  the 
scenario.  Range  differences  were  calculated  by  comparing 

the  reported  range  for  the  COI  by  the  participant  to  the 

actual  range  of  the  contact.  These  ranges  were  recorded  in 

thousands  of  yards  (Kyards)  .  Column  five  was  used  for  the 

analysis  of  range  differences  between  different  levels  of 
expertise  in  Appendices  M  through  P. 

The  range  +/-,  within  range,  over  or  under  variables 

in  columns  four,  six,  and  seven,  respectively,  were  also 

used  to  analyze  how  accurate  the  different  experts  were  in 

estimating  ranges  for  the  COI.  The  range  +/-  variable 

requires  the  participant  to  calculate  an  acceptable 

variability  in  range  estimates  for  the  COI.  A  relatively 
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large  range  envelope  would  be  an  advantage  over  one  with 
smaller  range  envelope.  For  example,  a  range  envelope  that 
is  four  miles  long  has  a  greater  probability  of 
encompassing  the  COI  than  a  range  envelope  that  is  one 
mile.  The  over  or  under  variable  represents  a 
participant's  response  that  was  an  over-estimated  range. 

4 .  Example  of  Data  Formatting 

A  reported  range  estimate  of  10.0  Kyards  (10,000)  with 
a  variable  range  +/-  equal  to  3.0  Kyards  (3,000)  at  time 
00:10:40  would  represent  a  total  range  envelope  between  7.0 
Kyards  (7,000)  and  13.0  Kyards  (13,000).  The  COI,  in  this 
example,  is  located  at  11.0  Kyards  (11,000)  from  ownship. 
The  calculated  range  difference  would  be  1.000  Kyards 
(1000) .  In  Appendices  M  through  P,  a  "1"  is  listed  for  the 
within  range  variable  because  the  COI  was  located  within 
the  range  envelope.  Since  the  reported  range  of  10,000 
yards  is  less  than  the  actual  range  of  11,000  yards  a  "0" 
is  listed  for  the  over  or  under  variable  because  the  COI 
the  range  estimate  was  an  under-estimated  range. 
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IV.  RESULTS  AND  ANALYSIS 


A.  OVERVIEW 

This  section  outlines  the  statistical  analyses 
performed  on  the  data.  Section  B  describes  the  formatting 
required  to  analyze  the  data.  Section  C  provides  the 
analysis  of  the  scenarios.  Table  1  is  an  example  of  how 
the  data  are  summarized  for  each  section's  analysis. 


Scenario  X 

P- 

CO/XO  DH  FT  Value 

Time  (minutes) 

Range  (Kyards) 

Within  Range 

Range  Envelope  (Kyards) 
Over  or  Under 

Table  1 .  Example  of  How  Data  are  Presented  for  each 

Scenario . 

Several  surveys  were  used  to  acquire  the  data  to  be 
analyzed.  Biographical  data  are  reported  in  Appendix  A. 
Appendix  B  includes  the  performance  data  on  all  four 
scenarios.  Appendix  C  includes  the  NASA  Task  Load  Index 
(TLX)  which  used  to  assess  the  workload  of  the 
participants . 

Hypothesis  1  states  there  is  a  difference  between  the 
groups  of  SMEs  with  respect  to  their  ability  to  estimate 
ranges  to  a  COI  when  presented  with  uncertain  data. 
Performance  of  the  three  groups  was  compared  on  the 
following  variables:  time  to  estimate  range,  range 
estimations,  the  proportion  of  ranges  that  were  within  a 
specified  range  envelope,  and  whether  or  not  a  range 
estimate  was  over-estimated  or  under-estimated. 
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B.  RESULTS 

1 .  Scenario  I 

Scenario  One  was  a  time  sensitive  scenario.  No 
differences  between  the  experts  were  noted.  Appendix  B 
describes  the  scenario  in  detail;  Appendix  K  contains  the 
TLX  and  personal  preferences  for  all  participants.  Table  2 
provides  a  summary  of  the  Scenario  One  data. 


Scenario  One 

CO/XO 

DH 

FT 

P- 

Value 

Time 

7.06 

9.067 

9.617 

Range 

1.310 

0.876 

1.731 

>  .05 

Within  Range 

5 

6 

3 

>  .05 

Range  Envelope 

0.88 

0.95 

1.025 

>  .05 

Over  or  Under 

4 

3 

3 

>  .05 

Table  2 .  Mean  Performance  Data  for  Three  Groups  of  Experts 

for  Scenario  One. 


a.  Time  Comparison  for  Groups  of  Experts 

A  test  for  equal  variance  revealed  that  the  data 
was  not  normally  distributed  (Bartlett's  Test:  test 

statistic  =  6.18,  p  <  .05) .  A  boxplot  in  Figure  11 

presents  variances  between  groups  of  experts.  The  CO/XO 
SMEs  completed  Scenario  One  fastest  while  reporting  the 
lowest  time  constraint  pressure  on  the  NASA  TLX.  The  CO/XO 
group  reported  a  mean  of  5.5  (out  of  10)  .  The  mean  time 
pressure  for  all  participants  was  6.9. 
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Figure  11.  Boxplot  of  Time  versus  Expert  for  Scenario  One 

b.  Range  Difference  Comparison  for  Groups  of 
Experts 

A  test  for  equal  variance  revealed  that  the  data 
was  normally  distributed  (Bartlett's  Test:  test  statistic  = 
3.51,  p  >  .05)  .  A  one-way  ANOVA  revealed  there  were  no 

significant  differences  between  expert  groups  regarding 
estimating  ranges  to  a  COI  F(2,29)=.23,  p  >  .05.  There  is 
one  range  difference  (highlighted  on  Appendix  M)  that  is 
three  standard  deviations  from  the  mean,  therefore,  is 
considered  an  outlier.  A  second  one-way  ANOVA  produced 
similar  results.  Expertise  was  not  a  factor  when 

estimating  ranges  to  a  COI,  F(2, 29)  =1 . 25,  p  >  .05. 

The  DH  SMEs  reported  COI  solutions  with  the  least 
amount  of  range  difference.  DH  experts  reported  ranges 
that  were  approximately  400  and  800  yards  more  accurate 
than  the  CO/XO  and  FT  experts,  respectively.  The  DH 
experts  also  reported  the  lowest  personal  experience  (5.1 
out  of  10)  and  confidence  in  solution  (1.4  out  of  4)  on 
Appendix  B' s  personal  traits  section.  See  Appendix  K  for 


63 


details.  The  overall  means  for  personal  experience  and 

confidence  in  solutions  were  6.0  and  2.0,  respectively. 

c.  Within  Range  Comparison  for  Groups  of 

Experts 

The  DH  and  CO/XO  experts  reported  range  envelopes 
that  contained  the  COI  50%  of  the  time  or  better.  The  FT 
experts  report  range  envelopes  that  contained  the  COI  30% 

of  the  time.  Regression  was  used  to  determine  if  there  was 
a  significant  difference  between  expert  groups  with  regard 
to  reporting  a  range  envelope  that  contained  the  COI . 
There  was  no  evidence  that  expertise  was  a  factor  in  range 
envelope  accuracy  (test  that  all  slopes  are  zero:  G  = 
0.809,  df  =  1,  p  =  0.368)  . 

d.  Range  Envelope  Comparison  for  Groups  of 

Experts 

A  one-way  ANOVA  revealed  there  was  no  significant 
difference  between  expert  groups  regarding  the  size  of 
range  envelope  estimates  for  the  COI  F(2,29)=.26,  p  >  .05. 

e.  Over  or  Under  Estimate  Comparison  for  Groups 
of  Experts 

The  DH  and  FT  expert  groups  over-estimated  the 
range  to  the  COI  30%  of  the  time  while  the  CO/XO  expert 
group  overestimated  the  range  to  the  COI  40%  of  the  time. 
Regression  was  used  to  determine  if  there  was  a  significant 
difference  between  expert  groups  regarding  reported  over¬ 
estimated  or  under-estimated  ranges  to  the  COI.  There  is 
no  evidence  that  expertise  was  a  factor  in  over-estimating 
or  under-estimating  ranges  to  a  COI  (test  that  all  slopes 
are  zero:  G  =  0.226,  df  =  1,  p  =  0.635) . 
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2. 


Scenario  II 


Scenario  Two  was  not  as  time  intensive  as  both 
Scenarios  One  and  Four.  No  differences  between  the  experts 
were  noted.  Appendix  B  describes  the  scenario  in  detail; 
Appendix  L  contains  the  TLX  and  personal  preferences  for 
all  participants.  Table  3  provides  a  summary  of  the 
Scenario  Two  data. 


Scenario  Two 

CO/XO  DH 

FT 

P- 

Value 

Time 

4.614 

6.047 

5.583 

>  .05 

Range 

4.355 

3.968 

4.786 

>  .05 

Within  Range 

2 

3 

1 

>  .05 

Range  Envelope 

2.49 

2.35 

2.1 

>  .05 

Over  or  Under 

0 

1 

0 

>  .05 

Table  3.  Mean  Performance  Data  for  Three  Groups  of  Experts 

for  Scenario  Two. 

a.  Time  Comparison  for  Groups  of  Experts 

A  test  for  equal  variance  revealed  that  the  data 
was  normally  distributed  (Bartlett's  Test:  test  statistic  = 
1.25,  p  >  .05)  .  A  one-way  ANOVA  revealed  there  were  no 

significant  differences  between  expert  groups  regarding  the 
length  of  time  it  took  for  them  to  generate  their  COI 
solutions  F(2,29)=.78,  p  >  .05.  The  CO/XO  SMEs  completed 

Scenario  Two  fastest  while  reporting  the  lowest  time 
constraint  pressure  on  the  NASA  TLX.  The  CO/XO  group 
reported  a  mean  of  5.6  (out  of  10) .  The  mean  time  pressure 
for  all  participants  was  5.7. 

b.  Range  Difference  Comparison  for  Groups  of 
Experts 

A  test  for  equal  variance  revealed  that  the  data 
was  normally  distributed  (Bartlett's  Test:  test  statistic  = 
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1.81,  p  >  .05).  A  one-way  ANOVA  revealed  there  were  no 

significant  differences  between  expert  groups  regarding 
estimating  ranges  to  a  COI  F(2,29)=.39,  p  >  .05. 

The  DH  SMEs  reported  COI  solutions  with  the  least 
amount  of  range  difference.  DH  experts  reported  ranges 
that  were  approximately  400  and  800  yards  more  accurate 
then  the  CO/XO  and  FT  experts,  respectively.  The  DH 

experts  also  reported  the  lowest  personal  experience  (5.5 
out  of  10)  and  confidence  in  solution  (2.1  out  of  4)  on 
Appendix  B' s  personal  traits  section.  See  Appendix  L  for 
details.  The  overall  means  for  personal  experience  and 
confidence  in  solutions  were  6.2  and  2.3,  respectively. 

c.  Within  Range  Comparison  for  Groups  of 

Experts 

The  DH  and  CO/XO  experts  reported  range  envelopes 
that  contained  the  COI  30%  and  20%  of  the  time, 

respectively.  The  FT  experts  report  range  envelopes  that 
contained  the  COI  10%  of  the  time.  Regression  was  used  to 
determine  if  there  was  a  significant  difference  between 
expert  groups  with  regard  to  reporting  a  range  envelope 
that  contains  the  COI.  Approximately  20%  of  the  total 
participants  reported  range  envelopes  that  contained  the 
COI.  There  was  no  evidence  that  expertise  was  a  factor  in 
range  envelope  accuracy  (test  that  all  slopes  are  zero:  G  = 
0.314,  df  =  1,  p  =  0.575)  . 

d.  Range  Envelope  Comparison  for  Groups  of 
Experts 

A  one-way  ANOVA  revealed  there  was  no  significant 
difference  between  expert  groups  regarding  the  size  of 

range  envelope  estimates  for  the  COI  F(2,29)=.28,  p  >  .05. 
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e.  Over  or  Under  Estimate  Comparison  for  Groups 
of  Experts 

Regression  was  used  to  determine  if  there  was  a 
significant  difference  between  expert  groups  regarding 
reported  over-estimated  or  under-estimated  ranges  to  the 
COI .  One  participant  over-estimated  the  range  to  the  COI . 
There  is  no  evidence  that  expertise  was  a  factor  in  over¬ 
estimating  or  under-estimating  ranges  to  a  COI  (test  that 
all  slopes  are  zero:  G  =  0.000,  df  =  1,  p  =  1.000) . 

3.  Scenario  III 

Scenario  Three  was  not  as  time  intensive  as  both 
Scenarios  One  and  Four.  One  difference  between  the  experts 
was  noted.  A  difference  between  DH  and  FT  SMEs  is  accuracy 
of  range  envelopes.  Appendix  B  describes  the  scenario  in 
detail;  Appendix  M  contains  the  TLX  and  personal 
preferences  for  all  participants.  Table  4  provides  a 
summary  of  the  Scenario  Three  data. 


Scenario  Three 

CO/XO  DH 

FT 

P- 

Value 

Time 

8.213 

8.344 

9.53 

>  .05 

Range 

1.603 

1.082 

1.654 

>  .05 

Within  Range 

3 

5 

1 

<  .05 

Range  Envelope 

0.965 

1.05 

0.9 

>  .05 

Over  or  Under 

3 

3 

2 

>  .05 

Table  4 .  Mean  Performance  Data  for  Three  Groups  of  Experts 

for  Scenario  Three. 

a.  Time  Comparison  for  Groups  of  Experts 

A  test  for  equal  variance  revealed  that  the  data 
was  normally  distributed  (Bartlett's  Test:  test  statistic  = 
.58,  p  >  .05)  .  A  one-way  ANOVA  revealed  there  were  no 

significant  differences  between  expert  groups  regarding  the 
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length  of  time  it  took  for  them  to  generate  their  COI 
solutions  F(2,28)=.45,  p  >  .05.  The  CO/XO  SMEs  completed 

Scenario  Three  fastest  while  reporting  the  lowest  time 
constraint  pressure  on  the  NASA  TLX.  The  CO/XO  group 
reported  a  mean  of  5.1  (out  of  10) .  The  mean  time  pressure 
for  all  participants  was  5.9. 

b.  Range  Difference  Comparison  for  Groups  of 
Experts 

A  test  for  equal  variance  revealed  that  the  data 
was  normally  distributed  (Bartlett's  Test:  test  statistic  = 
2.09,  p  >  .05)  .  A  one-way  ANOVA  revealed  there  were  no 

significant  differences  between  expert  groups  regarding 
estimating  ranges  to  a  COI  F(2,29)=.23,  p  >  .05. 

The  DH  SMEs  reported  COI  solutions  with  the  least 
amount  of  range  difference.  DH  experts  reported  ranges 
that  were  approximately  600  yards  more  accurate  than  the 
CO/XO  and  FT  experts.  The  DH  experts  also  reported  the 
lowest  personal  experience  (4.1  out  of  10)  and  confidence 
in  solution  (1.5  out  of  4)  on  Appendix  B' s  personal  traits 
section.  See  Appendix  M  for  details.  The  overall  means 
for  personal  experience  and  confidence  in  solutions  were 
5.4  and  1.9,  respectively. 

c.  Within  Range  Comparison  for  Groups  of 
Experts 

The  DH  experts  reported  range  envelopes  that 

contained  the  COI  50%  of  the  time  while  CO/XO  experts 

reported  range  envelops  that  contained  the  COI  30%  of  the 

time.  Regression  was  used  to  determine  if  there  was  a 

significant  difference  between  expert  groups  with  regard  to 

reporting  a  range  envelope  that  contains  the  COI .  There 
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was  no  evidence  that  expertise  was  a  factor  in  range 
envelope  accuracy  (test  that  all  slopes  are  zero:  G  = 

0.963,  df  =  1,  p  =  0.326),  however,  the  goodness-of-f  it 

tests  revealed  that  there  was  sufficient  evidence  for  the 
model  not  fitting  the  data  adequately.  Three  additional 
regressions  were  analyzed  with  regard  to  reporting  range 
envelopes  that  contain  the  COI . 

A  regression  between  expert  groups  one  and  two 
revealed  there  is  a  difference  regarding  range  envelope 
accuracy  (test  that  all  slopes  are  zero:  G  =  4.070,  df  =  1, 
p  =  0.044) .  A  regression  between  expert  groups  two  and 

three  revealed  there  is  no  difference  regarding  range 
envelope  accuracy  (test  that  all  slopes  are  zero:  G  = 

0.840,  df  =  1,  p  =  0.359)  .  A  regression  between  expert 
groups  one  and  three  revealed  there  is  no  difference 
regarding  range  envelope  accuracy  (test  that  all  slopes  are 
zero:  G  =  1.297,  df  =  1,  p  =  0.255) . 

d.  Range  Envelope  Comparison  for  Groups  of 

Experts 

A  one-way  ANOVA  revealed  there  was  no  significant 
difference  between  expert  groups  regarding  the  size  of 
range  envelope  estimates  for  the  COI  F(2,29)  =  .15,  p  >  .05. 

There  is  one  range  envelope  (highlighted  on  Appendix  M) 

that  is  three  standard  deviations  from  the  mean,  therefore, 
is  considered  an  outlier.  The  participant  who  recorded  the 
range  envelope  outlier  also  scored  a  range  difference  of 
3.962.  No  further  analysis  was  done  because  the  larger 
range  envelope  did  not  provide  any  benefit  to  the 
participants  within  range  results. 
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e.  Over  or  Under  Estimate  Comparison  for  Groups 
of  Experts 

All  three  experts  over-estimated  the  range  to  the 
COI  approximately  30%  of  the  time.  Regression  was  used  to 
determine  if  there  was  a  significant  difference  between 
expert  groups  regarding  reported  over-estimated  or  under¬ 
estimated  ranges  to  the  COI.  There  is  no  evidence  that 
expertise  was  a  factor  in  over-estimating  or  under¬ 
estimating  ranges  to  a  COI  (test  that  all  slopes  are  zero: 
G  =  0.257,  df  =  1,  p  =  0.613) . 

4 .  Scenario  IV 

Scenario  Four  was  a  time  sensitive  scenario.  One 
difference  between  the  experts  was  noted.  No  differences 
between  the  experts  were  noted.  Appendix  B  describes  the 
scenario  in  detail;  Appendix  N  contains  the  TLX  and 
personal  preferences  for  all  participants.  Table  5 
provides  a  summary  of  the  Scenario  Four  data. 


Scenario  Four 

CO/XO  DH 

FT 

P- 

Value 

Time 

4.844 

2.933 

4.747 

>  .05 

Range 

5.097 

3.772 

4.976 

>  .05 

Within  Range 

1 

4 

1 

>  .05 

Range  Envelope 

2.35 

2.17 

1.95 

>  .05 

Over  or  Under 

0 

2 

0 

>  .05 

Table  5.  Mean  Performance  Data  for  Three  Groups  of  Experts 

for  Scenario  Four. 

a.  Time  Comparison  for  Groups  of  Experts 

A  test  for  equal  variance  revealed  that  the  data 
was  normally  distributed  (Bartlett's  Test:  test  statistic  = 
2.96,  p  >  .05)  .  A  one-way  ANOVA  revealed  there  were  no 

significant  differences  between  expert  groups  regarding  the 
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length  of  time  it  took  for  them  to  generate  their  COI 
solutions  F(2,29)=  1.74,  p  >  .05.  The  DH  SMEs  completed 

Scenario  Four  fastest.  The  DH  group  reported  a  mean  of  7.7 
(out  of  10)  .  The  mean  time  pressure  for  all  participants 
was  7.7. 

b.  Range  Difference  Comparison  for  Groups  of 
Experts 

A  test  for  equal  variance  revealed  that  the  data 
was  normally  distributed  (Bartlett's  Test:  test  statistic  = 
1.43,  p  >  .05)  .  A  one-way  ANOVA  revealed  there  were  no 
significant  differences  between  expert  groups  regarding 
estimating  ranges  to  a  COI  F(2,29)=  1.09,  p  >  .05. 

The  DH  SMEs  reported  COI  solutions  with  the  least 
amount  of  range  difference.  DH  experts  reported  ranges 
that  were  approximately  1300  and  1200  yards  more  accurate 
then  the  CO/XO  and  FT  experts,  respectively.  The  DH 
experts  also  reported  the  lowest  personal  experience  (4.3 
out  of  10)  and  confidence  in  solution  (1.7  out  of  4)  on 
Appendix  B' s  personal  traits  section.  See  Appendix  N  for 
details.  The  overall  means  for  personal  experience  and 
confidence  in  solutions  were  5.7  and  2.0,  respectively. 

c.  Within  Range  Comparison  for  Groups  of 
Experts 

The  DH  experts  reported  range  envelopes  that 
contained  the  COI  40%  of  the  time  and  the  CO/XO  and  FT 
experts  reported  range  envelops  that  contained  the  COI  10% 
of  the  time.  Regression  was  used  to  determine  if  there  was 
a  significant  difference  between  expert  groups  with  regard 
to  reporting  a  range  envelope  that  contains  the  COI .  There 
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was  no  evidence  that  expertise  was  a  factor  in  range 
envelope  accuracy  (test  that  all  slopes  are  zero:  G  = 
2.532,  df  =  1,  p  =  .112)  . 

d.  Range  Envelope  Comparison  for  Groups  of 
Experts 

A  one-way  independent  ANOVA  indicates  there  were 
no  significant  differences  between  expert  groups  regarding 
the  size  of  range  envelope  estimates  for  the  COI 
F(2,29)=.22,  p  >  .05. 

e.  Over  or  Under  Estimate  Comparison  for  Groups 
of  Experts 

Regression  was  used  to  determine  if  there  was  a 
significant  difference  between  expert  groups  regarding 
reported  over-estimated  or  under-estimated  ranges  to  the 
COI .  Two  participants  over-estimated  the  range  to  the  COI . 
There  is  no  evidence  that  expertise  was  a  factor  in  over¬ 
estimating  or  under-estimating  ranges  to  a  COI  (test  that 
all  slopes  are  zero:  G  =  0.000,  df  =  1,  p  =  1.000) . 

5.  Experiment  Progression 

a.  Range  Difference  Comparison  as  the 
Experiment  Progressed 

The  analysis  in  this  section  does  not  consider  the 
participants.  The  participant  scores  were  combined  for 
each  scenario  progression.  A  test  for  equal  variance 
revealed  that  the  data  was  normally  distributed  (Bartlett's 
Test:  test  statistic  =  2.60,  p  >  .05)  .  A  one-way  ANOVA 

revealed  there  were  no  significant  differences  between 
scenarios  as  the  experiment  progressed  regarding  range 
estimations  F(3, 119) =. 40,  p  >  .05.  This  result  is  contrary 
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to  any  assumption  that  the  participants  range  estimation 
should  get  better  as  they  experience  more  AOU  scenarios. 
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V. 


SUMMARY,  DISCUSSION,  AND  RECOMMENDATIONS 


A.  SUMMARY 

There  is  no  significant  difference  between  the  expert 
groups  when  it  came  to  generating  range  solutions  to  a  COI . 
There  was  a  significant  difference  between  the  DH  and  FT 
expert  groups  within  range  estimates. 

B.  DISCUSSION 

1.  No  Range  Accuracy  Significant  Difference 

While  the  sample  of  DH  experts  in  the  current  study 
reported  more  accurate  range  estimates  than  the  other  two 
expert  groups,  this  difference  was  not  statistically 
significant.  It  is  possible  that  AOUs  do  not  provide 
enough  information  for  SMEs  to  develop  solutions  that  are 
significantly  different.  All  officers  receive  the  same  TMA 
training  during  their  careers.  Logically  the  CO/XO  SMEs 
should  report  more  accurate  ranges  to  a  COI  when  given 
similar  conditions  as  the  DH  and  ET  SMEs. 

Shanteau  has  done  extensive  research  in  the  behavior 
of  experts.  Experts  within  similar  domains  display  similar 
psychological  characteristics  and  have  a  sense  of  what  is 
relevant  or  irrelevant  (Shanteau,  1988).  Shanteau  also 
states  that  there  is  indeed  a  difference  between  mid-level 
and  advanced  in  that  the  advanced  experts  are  better  at 
evaluating  which  information  is  the  most  relevant 
(Shanteau,  1992b). 

No  differences  between  SMEs  were  observed  because  AOUs 
alone  do  not  provide  enough  information  to  develop 
solutions  that  are  representative  of  the  expert's  ability. 
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The  AOU  provides  a  minimal  amount  of  data  with  a  high 
degree  of  uncertainty.  The  solutions  generated  by  the  SMEs 
are  just  as  uncertain  as  the  data  from  which  they  were 
generated.  AOUs  that  are  accompanied  by  additional 
information  such  as  bearing  rate  and  bearings  would  be  a 
better  use  of  an  expert's  TMA  abilities. 

2.  Range  Envelope  Significant  Differences 

A  significant  difference  was  noted  between  the  DH  and 
FT  experts  regarding  the  ability  to  develop  range  envelopes 
that  contain  the  COI .  The  DHs  were  able  to  generate  ranges 
to  the  COI  that  were  800  yard  more  accurate  than  the  FT 
experts.  The  DHs  utilization  of  a  slightly  larger  range 
envelope  (1.63  Kyards)  when  compared  to  the  FT  experts 
(1.49Kyards)  along  with  their  more  accurate  range 
estimations  is  the  reason  for  the  significant  difference. 

C .  RECOMMENDATIONS 

1 .  Future  Decision  Aids 

The  results  of  the  current  study  demonstrate  that 
there  are  no  differences  between  SMEs  regarding  range 
accuracies  given  an  AOU  so  any  decision  aid  produced  would 
be  applicable  to  all  users,  regardless  of  their  expertise. 
The  information  obtained  in  the  post  scenario  questionnaire 
(Appendix  D)  is  invaluable  in  the  development  of  a  new 
decision  aid.  Two  examples  of  requested  information  that 
could  be  displayed  with  an  AOU  is  bearing  rate  and  AOU 
history.  Bearing  rates  are  a  key  ingredient  in 
understanding  relative  motion  between  a  COI  and  ownship. 
AOU  history  provides  a  track  of  the  COI's  relative  motion. 


76 


2. 


Future  Studies 


I  recommend  a  study  that  presents  a  COIs  bearing  rate, 
bearing,  AOU,  and  AOU  history  to  expert  groups  to  those 
used  in  the  current  study.  The  recommended  study  should 
have  more  than  ten  participants  in  each  expert  group.  The 
requirements  for  these  participants  should  meet  or  exceed 
the  requirements  used  in  the  current  study.  It  is 


suggested  that  " 

the  amount  of  information 

used 

does 

not 

reflect  degree 

of  expertise;  however. 

the 

type 

of 

information  used 

does"  (Shanteau,  1992b,  p. 

2)  . 

In 

this 

future  study  more  information  would  be  presented  to  three 
groups  of  SMEs  allowing  for  results  that  may  or  may  not 
support  the  findings  in  the  current  study. 
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APPENDIX  A:  BIOGRAPHICAL  DATA 


Biographical  Data.  Subject  # _ 

1 .  Age : 

2.  Education  Year  Degree  Major 

High  School : 

Under  graduate: 

Graduate : 

Post  graduate: 


3.  Navy  Experience 
Rank 

Years  in  the  Navy 
Time  at  sea  (Months) 

Time  since  last  sea  tour  (Months) 

4.  Combat  Systems 

List  the  combat  Systems  you  have  most  recently  used  Dates  used 


5.  Add  any  other  relevant  experience  you  feel  may  have  affected  your  results. 
For  example,  being  an  Instructor  or  having  a  lot  of  experience  in  high  density 
areas  while  being  submerged. 


6.  List  schools  attended  that  have  developed  your  TMA  skills. 


7 .  Are  you  Introverted  or  Extroverted? 

8.  Email  address 
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APPENDIX  B:  FOUR  SCENARIOS 


Scenario  1 

In  this  scenario  you  are  required  to  proceed  to  PD  in  an  expeditious  way.  The 
submarine  is  required  to  receive  the  next  broadcast  and  you  must  use  caution,  but  at  the 
same  time  proceed  as  quickly  as  possible.  Stop  the  scenario  once  you  are  ready  to 
proceed  to  PD. 

What  was  your  final  solution  for  the  COI? 

Range  +  or  -  ,  Bearing  +  or  -  ,  Course  +  or  - 

Using  a  range  between  1  and  10  (10  being  the  most),  to  answer  how  much  each  factor  weighed 
in  your  decision  making. 

Safety  of  ship _ (Do  not  want  to  sink  the  ship) 

Job  Satisfaction _ (You  have  trained  hard  so  show  it) 

Job  Security/  Promotion _ (Do  not  want  to  be  relieved  or  demoted  for  collision) 

Letting  down  the  team _ (Everyone  is  counting  on  you  to  make  a  good  decision) 

Mission  Accomplishment _ (Be  the  best  boat  on  the  waterfront) 

Personal  Satisfaction _ (You  do  everything  well,  even  taking  out  the  trash) 

Time  Constraint _ (The  ship  needs  to  get  to  PD  in  a  hurry) 

Personal  Experience _ (You  have  seen  this  stuff  a  million  times) 

How  confident  were  you  in  your  ability  to  compute  a  solution  given  the  AOU  data? 

I _ I _ I _ I _ I 

Not  Confident  Mildly  Confident  Very  Extremely 

Confident  Confident  Confident 
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Scenario  2 


In  this  scenario  you  are  required  to  proceed  to  PD.  Consider  this  trip  to  be  a 
routine  visit  to  receive  the  broadcast,  ventilate,  and  train.  The  submarine’s  only  threat  is 
collision  and  the  CO  tells  you  to  get  us  there  as  soon  as  you  feel  comfortable.  Stop  the 
scenario  once  you  are  ready  to  proceed  to  PD. 

What  was  your  final  solution  for  the  COI? 

Range  +  or  -  ,  Bearing  +  or  -  ,  Course  +  or  - 

Using  a  range  between  1  and  10  (10  being  the  most),  answer  how  much  each  factor  weighed  in 
your  decision  making. 

Safety  of  ship _ (Do  not  want  to  sink  the  ship) 

Job  Satisfaction _ (You  have  trained  hard  so  show  it) 

Job  Security/  Promotion _ (Do  not  want  to  be  relieved  or  demoted  for  collision) 

Letting  down  the  team _ (Everyone  is  counting  on  you  to  make  a  good  decision) 

Mission  Accomplishment _ (Be  the  best  boat  on  the  waterfront) 

Personal  Satisfaction _ (You  do  everything  well,  even  taking  out  the  trash) 

Time  Constraint _ (The  ship  can  take  more  time) 

Personal  Experience _ (You  have  seen  this  stuff  a  million  times) 

How  confident  were  you  in  your  ability  to  compute  a  solution  given  the  AOU  data? 

I _ I _ I _ I _ I 

Not  Confident  Mildly  Confident  Very  Extremely 

Confident  Confident  Confident 
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Scenario  3 


In  this  scenario  you  are  required  to  destroy  an  enemy  submarine.  The 
submarine  you  are  hunting  is  inferior  to  your  own,  you  have  multiple  torpedoes,  and  your 
sensors  are  much  better  than  their  sensors.  The  CO  wants  to  ensure  that  we  sink  the 
submarine  on  the  first  shot.  Stop  the  scenario  once  you  are  comfortable  with  the  COI 
solution  to  fire  a  torpedo. 

What  was  your  final  solution  for  the  COI? 

Range  +  or  -  ,  Bearing  +  or  -  ,  Course  +  or  - 

Using  a  range  between  1  and  10  (10  being  the  most),  answer  how  much  each  factor  weighed  in 
your  decision  making. 

Safety  of  ship _ (Do  not  want  to  be  shot  at) 

Job  Satisfaction _ (You  have  trained  hard  so  show  it) 

Job  Security/  Promotion _ (Do  not  want  to  be  relieved  or  demoted  for  collision) 

Letting  down  the  team _ (Everyone  is  counting  on  you  to  make  a  good  decision) 

Mission  Accomplishment _ (Be  the  best  boat  on  the  waterfront) 

Personal  Satisfaction _ (You  do  everything  well,  even  taking  out  the  trash) 

Time  Constraint _ (The  ship  can  take  more  time) 

Personal  Experience _ (You  have  seen  this  stuff  a  million  times) 

How  confident  were  you  in  your  ability  to  compute  a  solution  given  the  AOU  data? 

I _ I _ I _ I _ I 

Not  Confident  Mildly  Confident  Very  Extremely 

Confident  Confident  Confident 
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Scenario  4 


In  this  scenario  you  are  required  to  destroy  an  enemy  submarine.  The 
submarine  you  are  hunting  is  equivalent  to  your  own.  There  is  only  one  torpedo  left  and 
your  sensors  are  similar  to  their  sensors.  The  CO  wants  to  ensure  that  we  sink  the 
submarine  on  the  first  shot  and  also  to  not  get  shot  at  ourselves.  Stop  the  scenario  once 
you  are  comfortable  with  the  COI  solution  to  fire  a  torpedo. 

What  was  your  final  solution  for  the  COI? 

Range  +  or  -  ,  Bearing  +  or  -  ,  Course  +  or  - 

Using  a  range  between  1  and  10  (10  being  the  most),  answer  how  much  each  factor  weighed  in 
your  decision  making. 

Safety  of  ship _ (Do  not  want  to  sink  the  ship) 

Job  Satisfaction _ (You  have  trained  hard  so  show  it) 

Job  Security/  Promotion _ (Do  not  want  to  be  relieved  or  demoted  for  collision) 

Letting  down  the  team _ (Everyone  is  counting  on  you  to  make  a  good  decision) 

Mission  Accomplishment _ (Be  the  best  boat  on  the  waterfront) 

Personal  Satisfaction _ (You  do  everything  well,  even  taking  out  the  trash) 

Time  Constraint _ (The  ship  needs  to  make  a  decision  quick) 

Personal  Experience _ (You  have  seen  this  stuff  a  million  times) 

How  confident  were  you  in  your  ability  to  compute  a  solution  given  the  AOU  data? 

I _ I _ I _ I _ I 

Not  Confident  Mildly  Confident  Very  Extremely 

Confident  Confident  Confident 
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APPENDIX  C:  NASA  TASK  LOAD  INDEX 


We  are  interested  not  only  in  assessing  your  performance  but  also  the  experiences  you  had  during 
the  task.  In  the  most  general  sense  we  are  examining  the  “workload”  you  experienced.  The  factors  that 
influence  your  experience  of  workload  may  come  from  the  task  itself,  your  feelings  about  your  performance, 
how  much  effort  you  put  in,  or  the  stress  and  frustration  you  felt. 

Since  workload  is  something  experienced  by  each  person,  there  are  no  effective  “rulers”  that  can 
be  used  to  estimate  the  workload  of  different  activities.  One  way  to  find  out  about  workload  is  to  ask  people 
to  describe  the  feelings  they  experienced.  Because  workload  may  be  caused  by  many  different  factors,  we 
would  like  you  to  evaluate  several  of  them  individually  rather  than  lumping  them  into  a  single  global 
evaluation  of  workload.  Please  read  the  definitions  of  the  six  scales  carefully. 


•  Mental  Demand: 

•  Physical  Demand: 

•  Temporal  Demand: 

.  Effort: 

•  Performance: 

•  Frustration: 


whether  this  task  affects  a  user’s  attention,  brain,  and  focus, 
whether  this  task  affects  a  user’s  health,  makes  a  user  tired,  etc. 
whether  this  task  takes  a  lot  of  time  that  a  user  can’t  afford, 
whether  a  user  has  spent  a  lot  of  effort  for  this  task, 
whether  this  task  is  heavy  or  light  in  terms  of  workload, 
whether  this  task  makes  a  user  unhappy  or  frustrated. 


Now  you  will  perform  a  rating  task.  In  the  rating  task,  you  will  evalutate  the  task  by  marking  each  of 
the  6  rating  scales  at  the  point  which  matches  your  experience.  Each  scale  has  two  end  point  descriptors 
that  describe  the  scale.  Please  consider  your  responses  carefully  in  distinguishing  amoung  the  task 
conditions.  Consider  each  scale  individually. 


MENTAL  DEMAND 


Low  High 


PHYSICAL  DEMAND 


Low  High 

TEMPORAL  DEMAND 


Low  High 


EFFORT 


Low  High 


PERFORMANCE 


Low  High 


Low 


High 
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APPENDIX  D:  POST  SCENARIO  QUESTIONNAIRE 


Post  Scenario  Questionnaire.  Subject  # 


Please  consider  your  answers  carefully  and  mark  your  responses  on  the 
scales  provided.  Any  comments  you  have  will  be  useful  in  developing 
tools  and  displays  to  be  used  on  future  submarine  platforms.  Thank 
you  for  your  time  and  effort,  your  participation  is  very  much 
appreciated . 

1 .  Having  a  graphical  representation  of  the  COI  range  estimation  made 
it  easier  to  compute  a  solution  range . 


Strongly  Disagree  Neither  Agree  Agree  Strongly 

Disagree  nor  Disagree  Agree 

Please  explain  your  rating; 


2 .  How  useful  were  the  AOUs  in  helping  you  compute  solutions  for  the 
COI? 


Not  Somewhat 

Useful  Useful 

Please  explain  your  rating; 


Moderately  Very  Extremely 

Useful  Useful  Useful 


3 .  Did  you  find  that  there  was  a  difference  in  using  the  AOU  with 
regard  to  the  two  different  tasks  (Navigation  and  Targeting) ? 
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4 .  Compared  with  past  TMA  experiences ,  the  displayed  AOU  reduced  the 
time  required  to  compute  the  solution. 


Strongly  Disagree  Neither  Agree  Agree  Strongly 

Disagree  nor  Disagree  Agree 

Please  explain  your  rating; 


5.  What  factors,  in  your  mind,  contribute  to  making  these  tasks 
difficult?  (PI  ease  list  all  factors  and  how  each  one  made  the 

different  tasks  hard  to  perform.) 

1 . _ 

2  . _ 

3. _ 

4  . _ 

5. 


6.  During  the  targeting  scenario,  with  regard  to  your  contact 
solution,  I  considered  the  torpedoes  abilities  to  detect  a  threat  even 
though  the  solution  may  have  been  inaccurate. 


Strongly  Disagree  Neither  Agree  Agree  Strongly 

Disagree  nor  Disagree  Agree 

Please  explain  your  rating; 


7 .  During  the  navigation  scenarios  what  was  your  limiting  factors  that 
determined  when  you  would  go  to  PD?  (Range,  contact  course,  time, 
other) 
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8 .  It  was  easy  to  understand  the  meaning  of  the  information  presented 
in  the  displayed  AOU. 


Strongly  Disagree  Neither  Agree  Agree  Strongly 

Disagree  nor  Disagree  Agree 

Please  explain  your  rating; 


9 .  I  am  highly  confident  in  my  abilities  to  conduct  TMA  in  both 
targeting  and  navigation  scenarios  in  the  fleet. 


Strongly  Disagree  Neither  Agree  Agree  Strongly 

Disagree  nor  Disagree  Agree 

Please  explain  your  rating; 


1 0 .  How  much  did  your  prior  training  experience  help  you  in 
decision  making  today? 

I _ I _ I _ I _ I 

Very  Little  Somewhat  Moderately  A  lot  Great  Extent 

Please  explain  your  rating; 


11.  My  experience  with  US  torpedoes  affected  my  decision  making 
ability  regarding  contact  solutions  before  firing. 


Strongly  Disagree  Neither  Agree  Agree  Strongly 

Disagree  nor  Disagree  Agree 

Please  explain  your  rating; 
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12.  Please  give  us  any  additional  feedback  you  feel  would  be 
valuable . 
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APPENDIX  E:  EXPERIMENTAL  PROTOCOL 


Experimental  Protocol:  Script  in  green  was  spoken  aloud  to  each  participant. 


Prior  to  subject  entering: 

Arrange  consent  forms,  questionnaire  ready, 
computer  on,  camera  ready,  program  ready. 

“This  is  an  unclassified  experiment  based  on  the 
problem  of  predicting  uncertainty.  We  will  be  using 
four  scenarios  chosen  in  a  random  order.  Two  of  the 
scenarios  will  be  tactical  and  two  will  be  navigational 
in  nature  and  they  will  all  be  different  with  the 
exception  of  the  mission  requirements.  1  will  explain 
the  direction  for  each  scenario  before  we  start  each 
one.  The  instructions  that  apply  to  one  scenario  may 
not  apply  to  another.” 

“After  each  scenario  you  will  complete  one  page  of 
the  questionnaire  as  well  as  a  workload  questionnaire. 
After  the  last  scenario  you  will  be  asked  to  complete 
the  Post-scenario  and  bio  questions.” 

“The  ellipses  give  you  two  areas  of  uncertainty.  These 
are  66%,  and  95%  containment.  They  will  be  color 
coded  differently  for  each  scenario,  but  the  percent 
containment  will  remain  the  same. 

The  solution  screen  will  update  itself 
approximately  every  10sec. 

Range  Rings  are  established  every  5000 
yards. 

Ownship  data  will  be  displayed  on  the  right 
hand  side  of  the  screen  for  your  convenience. 
COI  data  is  also  available  on  the  right  hand 
side  of  the  screen.  You  can  consider  this 
information  to  be  the  on  watch  FTOW’s 
solution  using  the  Fire  Control  computer.  It  is 
not  truth  but  is  available  for  your  use. 

Once  you  ‘click’  the  stop  button  at  the  bottom 
of  the  screen  the  picture  will  disappear.  At 
this  point  you  will  be  required  to  fill  out  the 
appropriate  forms. 

There  will  be  a  video  recorder  observing  your 
actions 

Do  you  have  any  questions  for  me?  If  you 
have  any  questions  during  the  scenario, 
please  ask.  “ 

Nav:  This  is  a  navigation  scenario.  The  goal  is  to 
understand  the  situation  well  enough  to  come  to  PD. 
Hit  “Stop”  when  you  are  ready  to  come  to  PD.  Your 
submarine  does  not  need  to  be  on  the  best  course  to 
proceed  to  PD. 

Tarqetinq:  This  is  an  approach  and  attack  scenario. 
The  goal  is  to  localize  the  COI  well  enough  to  shoot. 
Hit  “stop”  when  you  know  the  solution.  You  do  not 
need  to  be  in  a  firing  position. 

“Your  main  focus  will  be  range  and  bearing  to  the 
COI.  The  COI  course  is  also  requested  but  it  is 
understood  to  be  extremely  difficult  for  an  accurate 
course  using  the  AOU.  The  order  of  importance  in  the 
scenarios  will  be  range  then  bearing  and  a  course  if 
you  feel  comfortable  with  one.  Let  us  begin” 

Administer  each  scenario  and  record  the  results. 

Remember  to  thank  them  for  their  time  and  to  ask  if 
they  have  any  questions 
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APPENDIX  F:  CONSENT  FORM 


Naval  Postgraduate  School 
Consent  to  Act  as  a  Research  Participant 

How  to  Display  Uncertainty  in  a  Complex  and  Crowded  Submarine  Environment 

You  are  being  asked  to  participate  in  a  research  study.  Before  you  give  your  consent  to 
volunteer,  it  is  important  that  you  read  the  following  information  and  ask  as  many 
questions  as  necessary  to  be  sure  you  understand  what  you  will  be  asked  to  do. 

Investigator:  James  Prouty 

Purpose  of  the  Study:  This  study  is  designed  to  evaluate  three  different  levels  of  expertise  in 
submarine  subject  matter  experts.  Specifically  these  experts  will  be  evaluated  on  their 
generated  contact  solutions  based  on  their  use  of  displays  associated  with  target  motion 
analysis. 


Description  of  the  Study:  This  study  will  explore  and  empirically  evaluate  new  ways  to 
graphically  represent  the  Area  of  Uncertainty  (AOU)  for  a  submerged  contact.  I  will  administer 
scenarios  to  senior  officers,  junior  officers,  and  fire  control  technicians  who  are  submarine 
subject  matter  experts.  By  using  the  contact  solutions  generated  by  the  participants,  and  a 
survey  designed  to  collect  their  thought  processes,  I  will  investigate  the  different  logistical 
strategies  employed  by  the  three  groups. 

Risks  or  Discomforts:  Respondents  range  solutions  will  be  published  in  a  masters  thesis 
distributed  by  the  Naval  Postgraduate  School  and  analysis  of  the  responses  will  be  used  to 
draw  conclusions  in  the  aforementioned  thesis  research  paper.  Anonymity  is  guaranteed  to  the 
participants  in  the  experiment,  and  at  no  time  will  your  name  be  published  along  with  your 
answers.  Only  the  thesis  researchers  will  see  the  survey  responses  with  respondent  names 
included. 

Benefits  of  the  Study:  Your  participation  in  this  study  will  aid  in  research  regarding  the 
effectiveness  and  possible  changing  of  displays  in  the  submarine  environment. 

Possible  benefits  of  your  participation  in  this  study  are  an  improved  range  display  that 
may  be  generated  as  a  result  of  this  research. 

Confidentiality:  Names  are  only  linked  to  data  by  anonymous  codes.  Only  group,  or  otherwise 
anonymous,  data  will  be  reported.  Confidentiality  will  be  maintained  to  the  extent  allowed  by 
law. 

Incentives  to  Participate:  Participation  in  this  study  is  voluntary  and  you  will  not  be  paid  to 
participate  in  this  study. 
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Voluntary  Nature  of  Participation:  Participation  in  this  study  is  voluntary.  Your  choice  of 
whether  or  not  to  participate  will  not  influence  your  future  relations  with  the  Naval 
Postgraduate  School.  If  you  decide  to  participate,  you  are  free  to  withdraw  your  consent 
and  to  stop  your  participation  at  any  time  without  penalty  or  loss  of  benefits  to  which  you 
are  allowed. 

Questions  about  the  Study:  If  you  have  any  questions  about  the  research  now,  please  ask. 
If  you  have  questions  later  about  the  research,  you  may  contact: 


James  Prouty 
(831)392-5397 
irprouW@nps.edu 


Prof.  Susan  Hutchins 
(831)  656-3768 
shutchins@nps.edu 


If  you  have  any  questions  about  your  rights  as  a  participant  in  this  study,  you  may 
contact  the  NPS-IRB  Administrative  Official: 


Ms.  Danielle  Kuska 
(831)  656-2099 

dkuska@nps.edu 


Consent  to  Participate:  The  Naval  Postgraduate  School  -  Institutional  Review  Board  has 
approved  this  consent  form,  as  signified  by  the  Board's  stamp.  The  consent  form  must 
be  reviewed  annually  and  expires  on  the  date  indicated  on  the  stamp. 

Your  signature  below  indicates  that  you  have  read  the  information  in  this  document  and 
have  had  a  chance  to  ask  any  questions  you  have  about  the  study.  Your  signature  also 
indicates  that  you  agree  to  be  in  the  study  and  have  been  told  that  you  can  change  your 
mind  and  withdraw  your  consent  to  participate  at  any  time.  You  have  been  given  a  copy 
of  this  consent  form.  You  have  been  told  that  by  signing  this  consent  form  you  are  not 
giving  up  any  of  your  legal  rights. 

I  FULLY  UNDERSTAND  THAT  I  AM  MAKING  A  DECISION  WHETHER  OR  NOT  TO 
PARTICIPATE.  MY  SIGNATURE  INDICATES  THAT  I  HAVE  DECIDED  TO  PARTICIPATE, 
HAVING  READ  THE  INFORMATION  PROVIDED  ABOVE. 


Name  of  Participant  (please  print) 


Signature  of  Participant  Date 


Signature  of  Investigator  Date 

Subject#  Date 
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APPENDIX  G.  RECORDED  NASA  TASK  LOAD  INDEX  AND 
PERSONAL  TRAITS  FOR  SCENARIO  ONE 


Overall 

Mean 

2.6 

o 

CO 

1.9 

2.0 

1.9 

1.8 

9.0 

5.9 

4.7 

6.9 

8.7 

6.6 

6.8 

6.0 

2.0 

Safety 

Conf id 

Mental 

Physical 

Temporal 

of 

Job 

Job 

Let 

Mission 

Personal 

Time 

Personal 

in 

Sub  # 

Demand 

Demand 

Demand 

Effort 

Perform 

Frustrate 

Ship 

Sat 

Sec 

Down 

Accom 

Satisfy 

Constr 

Exper 

Solution 

3 

2.8 

0.4 

0.6 

1  6 

1.3 
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10 

5 

5 

10 

10 

7 
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APPENDIX  H.  RECORDED  NASA  TASK  LOAD  INDEX  AND 
PERSONAL  TRAITS  FOR  SCENARIO  TWO 
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APPENDIX  I.  RECORDED  NASA  TASK  LOAD  INDEX  AND 
PERSONAL  TRAITS  FOR  SCENARIO  THREE 
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APPENDIX  J.  RECORDED  NASA  TASK  LOAD  INDEX  AND 
PERSONAL  TRAITS  FOR  SCENARIO  FOUR 
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1.9 

8.3 

8.1 

4.6 

9.5 

9.6 

7.9 

8.7 

6.9 

2.1 

Safety 

Conf id 

Mental 

Physical 

Temporal 

of 

Job 

Job 

Let 

Mission 

Personal 

Time 

Personal 

in 

Sub  # 

Demand 

Demand 

Demand 

Effort 

Perform 

Frustrate 

Ship 

Sat 

Sec 

Down 

Accom 

Satisfy 

Constr 

Exper 

Solution 

1 

2  7 

1.2 

1.5 

1.5 

1  4 

2.5 

10 

8 

8 

9 

9 

8 

7 

6 

2.9 

2 

2  4 

0.4 

1  4 

2 

1.6 

2.2 

8 

7 

3 

8 

8 

10 

7 

3 

1 

5 

3.6 

2 

3 

3 

3 

2.5 

10 

6 

4 

5 

7 

8 

4 

1 

2 

6 

3.2 

0.7 

1.8 

2.3 

2.5 

1.6 

9 

1 

1 

6 

9 

5 

9 

1 

1 

13 

3.4 

1  6 

2.5 

3.2 

3 

3 

3 

3 

1 

6 

10 

7 

10 

5 

1 

18 

3 

0.6 

2.6 

3 

2.6 

2.8 

8 

7 

4 

7 

7 

7 

7 

4 

2 

19 

1.9 

0.3 

1.3 

1.3 

1.3 

3.6 

5 

6 

2 

9 

10 

8 

8 

8 

1 

21 

1.8 

1.9 

1  7 

1  6 

2 

1.8 

10 

7 

4 

6 

8 

7 

10 

6 

2 

25 

1 

1  4 

4 

2.8 

2.6 

1.6 

10 

5 

4 

7 

8 

8 

7 

5 

2.5 

30 

3 

1  6 

1.8 

2.9 

2.3 

1.9 

8 

8 

5 

8 

10 

7 

8 

4 

1.7 

Expert  Two 

Mean 

2.6 

1.2 

2.2 

2 . 4 

2.2 

2 . 4 

8.1 

5.8 

3.6 

7.1 

8.6 

7.5 

7.7 

4.3 

1.7 

Safety 

Conf id 

Mental 

Physical 

Temporal 

of 

Job 

Job 

Let 

Mission 

Personal 

Time 

Personal 

in 

Sub  # 

Demand 

Demand 

Demand 

Effort 

Perform 

Frustrate 

Ship 

Sat 

Sec 

Down 

Accom 

Satisfy 

Constr 

Exper 

Solution 

16 

2 

1 

3 

1 

2 

2 

10 

6 

6 

10 

10 

8 

2 

8 

3 

17 

3.8 

1  6 

3.1 

2  7 

2  4 

1.8 

10 

7 

10 

10 

10 

7 

7 

7 

3 

20 

1.6 

0.1 

1  4 

1  4 

1.2 

3.4 

8 

6 

1 

6 

8 

7 

7 

8 

2.5 

22 

3 

1.2 

1.8 

3 

2.8 

2  4 

10 

8 

4 

8 

9 

9 

6 

3 

2 

23 

3.4 

0.4 

0.8 

2 

1.2 

0.2 

3 

5 

1 

3 

10 

6 

8 

6 

2 

24 

2.2 

1.8 

0.7 

2  4 

3.3 

2  1 

10 

8 

4 

8 

8 

8 

7 

5 

1.9 

26 

2 

0.2 

2.6 

2  4 

2.3 

3 

10 

2 

2 

3 

8 

3 

8 

1 

1 

27 

3.5 

0.2 

1.6 

1.6 

3.2 

1.5 

10 

5 

3 

9 

9 

7 

7 

10 

2.7 

28 

2 

o 

CO 

2 

0.7 

1.6 

1  6 

10 

4 

3 

7 

8 

4 

5 

4 

2 

29 

1 

0 

0.6 

1 

1.5 

1 

10 

1 

1 

1 

7 

7 

8 

8 

3 

Expert  Three 

Mean 

2.5 

0.7 

1.8 

1.8 

2.2 

1.9 

9.1 

5.0 

3.5 

6.5 

8.7 

6.6 

6.5 

6 

2.3 
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APPENDIX  K.  SCENARIO  ONE  RAW  DATA 


Expert 

Time 

Range 

Range  +/- 

Range 

Difference 

Within  Range 

Over  or  Under 

1 

14.500 

3.0 

1.5 

3.041 

0 

0 

1 

11.500 

4.0 

1.0 

2.892 

0 

0 

1 

5.167 

3.3 

0.8 

0.521 

1 

1 

1 

7.167 

3.5 

1.0 

0.771 

1 

0 

1 

7.000 

3.5 

0.5 

0.641 

0 

0 

1 

11.167 

2.0 

1.5 

5.381* 

0 

0 

1 

12.667 

3.5 

1.0 

3.083 

0 

0 

1 

6.167 

6.0 

1.0 

2.623 

0 

1 

1 

6.000 

4.0 

0.5 

0.746 

0 

1 

1 

14.833 

4.7 

1.5 

1.262 

1 

0 

2 

7.000 

4.0 

1.0 

0.011 

1 

0 

2 

9.667 

2.0 

0.5 

4.284 

0 

0 

2 

8.000 

4.0 

1.0 

0.932 

1 

0 

2 

14.833 

5.0 

0.5 

0.962 

0 

0 

2 

14.833 

4.5 

1.0 

1.462 

0 

0 

2 

15.000 

5.0 

2.0 

0.962 

1 

0 

2 

5.000 

4.0 

0.2 

1.335 

0 

1 

2 

6.167 

3.0 

1.0 

0.377 

1 

0 

2 

5.667 

4.2 

1.3 

1.068 

1 

1 

2 

4.500 

3.2 

1.0 

0.777 

1 

1 

3 

7.500 

2.5 

1.0 

2.034 

0 

0 

3 

6.300 

2.5 

1.5 

1.002 

1 

0 

3 

4.667 

4.5 

0.5 

2.053 

0 

1 

3 

7.667 

3.5 

1.2 

1.166 

1 

0 

3 

9.000 

3.8 

0.5 

1.939 

0 

0 

3 

8.000 

6.0 

1.5 

1.068 

1 

1 

3 

5.167 

3.8 

0.4 

1.021 

0 

1 

3 

5.000 

2.5 

0.2 

0.165 

1 

0 

3 

7.300 

4.5 

1.0 

0.098 

1 

1 

3 

10.000 

4.0 

1.0 

2.557 

0 

0 

*  Note:  This  value  is  greater  than  three  standarc. 
deviations  from  the  mean. 
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APPENDIX  L.  SCENARIO  TWO  RAW  DATA 


Expert 

Time 

Range 

Range  +/- 

Range  Difference 

Within  Range 

Over  or  Under 

1 

8.000 

12.0 

3.0 

4.012 

0 

0 

1 

2.667 

10.0 

3.0 

4.967 

0 

0 

1 

2.500 

10.0 

2.0 

4.804 

0 

0 

1 

7.000 

11.0 

3.0 

4.560 

0 

0 

1 

6.167 

12.0 

2.0 

3.237 

0 

0 

1 

5.667 

7.0 

2.0 

8.275 

0 

0 

1 

8.500 

12.0 

2.0 

4.259 

0 

0 

1 

5.667 

10.0 

1.0 

5.275 

0 

0 

1 

1.833 

7.5 

1.0 

6.555 

0 

0 

1 

7.833 

14.0 

2.0 

1.918 

1 

0 

2 

4.000 

7.5 

2.0 

8.017 

0 

0 

2 

10.300 

15.0 

3.0 

2.650 

1 

0 

2 

6.667 

15.0 

5.0 

0.390 

1 

0 

2 

3.500 

8.0 

0.5 

7.472 

0 

0 

2 

4.667 

12.0 

2.0 

3.413 

0 

0 

2 

11.833 

16.0 

2.0 

3.382 

0 

0 

2 

2.833 

10.0 

1.0 

5.061 

0 

0 

2 

8.000 

12.0 

2.0 

4.012 

0 

0 

2 

5.833 

15.5 

4.0 

0.225 

1 

1 

2 

2.833 

10.0 

2.0 

5.061 

0 

0 

3 

2.667 

12.0 

4.0 

2.967 

1 

0 

3 

6.500 

14.0 

1.0 

1.267 

0 

0 

3 

6.167 

11.0 

3.0 

4.237 

0 

0 

3 

4.000 

9.0 

2.6 

6.517 

0 

0 

3 

9.000 

13.0 

3.0 

3.573 

0 

0 

3 

4.300 

8.5 

0.3 

6.977 

0 

0 

3 

2.500 

9.0 

2.0 

5.804 

0 

0 

3 

2.667 

12.0 

2.0 

2.967 

0 

0 

3 

5.667 

10.0 

2.0 

5.275 

0 

0 

3 

2.667 

11.0 

5.0 

3.967 

1 

0 
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APPENDIX  M.  SCENARIO  THREE  RAW  DATA 


Expert 

Time 

Range 

Range  +/- 

Range  Difference 

Within  Range 

Over  or 
Under 

1 

5.167 

4.0 

0.5 

1.221 

0 

1 

1 

10.000 

3.4 

2.0 

3.157 

0 

0 

1 

9.500 

4.0 

1.5 

2.147 

0 

0 

1 

15.000 

4.5 

1.0 

1.424 

0 

0 

1 

5.833 

3.5 

0.5 

0.368 

1 

1 

1 

7.300 

3.0 

0.5 

1.402 

0 

0 

1 

12.667 

4.0 

1.0 

2.583 

0 

0 

1 

14.667 

5.0 

1.0 

1.041 

0 

0 

1 

7.667 

3.0 

0.5 

1.666 

0 

0 

1 

7.500 

3.0 

0.5 

1.534 

0 

0 

2 

6.300 

2.5 

1.0 

1.002 

0 

0 

2 

6.000 

3.0 

1.0 

0.254 

1 

0 

2 

13.000 

4.5 

1.0 

1.985 

0 

0 

2 

10.833 

4.0 

0.5 

3.106 

0 

0 

2 

15.000 

5.5 

1.0 

0.424 

1 

0 

2 

15.000 

2.0 

3.0* 

3.962 

0 

0 

2 

6.667 

3.0 

1.0 

0.882 

1 

0 

2 

5.833 

3.5 

0.5 

0.368 

1 

1 

2 

5.167 

4.0 

0.5 

1.221 

0 

1 

2 

6.300 

4.0 

1.0 

0.498 

1 

1 

3 

7.667 

3.6 

1.6 

1.066 

1 

0 

3 

9.000 

3.5 

0.5 

2.239 

0 

0 

3 

9.500 

4.0 

1.0 

2.147 

0 

0 

3 

11.167 

4.0 

1.0 

3.381 

0 

0 

3 

5.167 

4.5 

0.5 

1.721 

0 

1 

3 

6.833 

3.0 

1.5 

1.011 

1 

0 

3 

4.833 

4.0 

1.0 

1.445 

0 

1 

3 

15.000 

6.0 

2.0 

0.038 

1 

1 

3 

6.300 

2.3 

0.3 

1.202 

0 

0 

3 

6.667 

2.1 

0.3 

1.782 

0 

0 

*  Note:  This  value  is  greater  than  three  standard 
deviations  from  the  mean. 
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APPENDIX  N.  SCENARIO  FOUR  RAW  DATA 


Expert 

Time 

Range 

Range  +/- 

Range  Difference 

Within  Range 

Over  or 
Under 

1 

5.167 

4.0 

0.5 

1.221 

0 

1 

1 

10.000 

3.4 

2.0 

3.157 

0 

0 

1 

9.500 

4.0 

1.5 

2.147 

0 

0 

1 

15.000 

4.5 

1.0 

1.424 

0 

0 

1 

5.833 

3.5 

0.5 

0.368 

1 

1 

1 

7.300 

3.0 

0.5 

1.402 

0 

0 

1 

12.667 

4.0 

1.0 

2.583 

0 

0 

1 

14.667 

5.0 

1.0 

1.041 

0 

0 

1 

7.667 

3.0 

0.5 

1.666 

0 

0 

1 

7.500 

3.0 

0.5 

1.534 

0 

0 

2 

6.300 

2.5 

1.0 

1.002 

0 

0 

2 

6.000 

3.0 

1.0 

0.254 

1 

0 

2 

13.000 

4.5 

1.0 

1.985 

0 

0 

2 

10.833 

4.0 

0.5 

3.106 

0 

0 

2 

15.000 

5.5 

1.0 

0.424 

1 

0 

2 

15.000 

2.0 

3.0 

3.962 

0 

0 

2 

6.667 

3.0 

1.0 

0.882 

1 

0 

2 

5.833 

3.5 

0.5 

0.368 

1 

1 

2 

5.167 

4.0 

0.5 

1.221 

0 

1 

2 

6.300 

4.0 

1.0 

0.498 

1 

1 

3 

7.667 

3.6 

1.6 

1.066 

1 

0 

3 

9.000 

3.5 

0.5 

2.239 

0 

0 

3 

9.500 

4.0 

1.0 

2.147 

0 

0 

3 

11.167 

4.0 

1.0 

3.381 

0 

0 

3 

5.167 

4.5 

0.5 

1.721 

0 

1 

3 

6.833 

3.0 

1.5 

1.011 

1 

0 

3 

4.833 

4.0 

1.0 

1.445 

0 

1 

3 

15.000 

6.0 

2.0 

0.038 

1 

1 

3 

6.300 

2.3 

0.3 

1.202 

0 

0 

3 

6.667 

2.1 

0.3 

1.782 

0 

0 
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